Light Field Display System for Sporting Events

ABSTRACT

A light field (LF) display system for displaying holographic content (e.g., a holographic sporting event or holographic content to augment a holographic sporting event) to viewers in an arena. The LF display system in the arena includes LF display modules tiled together to form an array of LF modules. The array of LF modules create a holographic object volume for displaying the holographic content in the arena. The array of LF modules displays the holographic content to viewers in the viewing volumes. The LF display system can be included in a LF sporting event network. The LF sporting event network allows holographic content to be created at one location and presented at another location. The LF sporting event network includes a network system to manage the digital rights of the holographic sporting event content.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to International Application Nos.PCT/US2017/042275, PCT/US2017/042276, PCT/US2017/042418,PCT/US2017/042452, PCT/US2017/042462, PCT/US2017/042466,PCT/US2017/042467, PCT/US2017/042468, PCT/US2017/042469,PCT/US2017/042470, and PCT/US2017/042679, all of which are incorporatedby reference herein in their entirety.

BACKGROUND

The present disclosure relates to presenting sporting events in anarena, and specifically relates to light field display systems forpresenting sporting events in an arena.

Traditionally, arenas (e.g., baseball stadiums, basketball arenas, etc.)are configured to allow viewers (e.g., fans, crowds, patrons, etc.)attending the area to view a sporting event (e.g., a baseball game, atennis match, etc.) in real-time. Unfortunately, in some cases, hostinga sporting event in an arena limits the ability of a viewer who wants toview the sporting event from doing so. For example, the sporting eventmay be sold out, may be at an inconvenient time, or may be located farfrom the viewer. Sometimes sporting events are recorded and,subsequently, reproduced on two-dimensional surfaces such as moviescreens or televisions, but these reproductions hardly reproduce theatmosphere and excitement present in an arena of a live sporting event.As such, presentation spaces configured to present sporting events suchthat viewers can perceive the sporting event as if they were at the livesporting event in an arena would be beneficial.

SUMMARY

A light field (LF) display system for displaying holographic content ofa sporting event in a presentation space (e.g., an area, a home theater,a public venue such as a bar, etc.). The LF display system includes LFdisplay modules that form a surface (e.g., a court, a floor, atable-top, etc.) in the presentation space, the LF display modules eachhave a display area and are tiled together to form a seamless displaysurface that has an effective display area that is larger than thedisplay area. The LF display modules display holographic content of asporting event from a sporting volume such that viewers in thepresentation space can perceive the sporting event.

In some embodiments, the holographic content of a sporting event may bea reproduction of a sporting event occurring concurrently at anotherpresentation space, created for display in the presentation space by acontent creation system, and/or accessed for display in the presentationspace from a data store. The holographic content can be managed by anetwork system responsible for managing the digital rights of theholographic content. For example, viewers in the presentation space maypay a transaction fee to access holographic content for display in thepresentation space.

In some embodiments, the LF display system includes a tracking systemand/or a viewer profiling system. The tracking system and profilingsystem can monitor and store characteristics of viewers in thepresentation space, a viewer profile describing a viewer, and/orresponses of viewers to the holographic content in the presentationspace. The holographic content created for display in a presentationspace can be based on any of the monitored or stored information.

In some embodiments, a user may interact with the holographic content,and the interaction can act as input for the holographic contentcreation system. For example, in some embodiments, some or all of the LFdisplay system includes a plurality of ultrasonic speakers. Theplurality of ultrasonic speakers are configured to generate a hapticsurface that coincides with at least a portion of the holographiccontent. The tracking system is configured to track an interaction of auser with the holographic object (e.g., via images captured by imagingsensors of the LF display modules and/or some other cameras). And the LFdisplay system is configured to provide to create holographic contentbased on the interaction.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a light field display module presenting aholographic object, in accordance with one or more embodiments.

FIG. 2A is a cross section of a portion of a light field display module,in accordance with one or more embodiments.

FIG. 2B is a cross section of a portion of a light field display module,in accordance with one or more embodiments.

FIG. 3A is a perspective view of a light field display module, inaccordance with one or more embodiments.

FIG. 3B is a cross-sectional view of a light field display module whichincludes interleaved energy relay devices, in accordance with one ormore embodiments.

FIG. 4A is a perspective view of portion of a light field display systemthat is tiled in two dimensions to form a single-sided seamless surfaceenvironment, in accordance with one or more embodiments.

FIG. 4B is a perspective view of a portion of light field display systemin a multi-sided seamless surface environment, in accordance with one ormore embodiments.

FIG. 4C is a top-down view of a light field display system with anaggregate surface in a winged configuration, in accordance with one ormore embodiments.

FIG. 4D is a side view of a light field display system with an aggregatesurface in a sloped configuration, in accordance with one or moreembodiments.

FIG. 4E is a top-down view of a light field display system with anaggregate surface on a front wall of a room, in accordance with one ormore embodiments.

FIG. 4F is a side view of a side view of a LF display system with anaggregate surface on the front wall of the room, in accordance with oneor more embodiments.

FIG. 5A is a block diagram of a light field display system, inaccordance with one or more embodiments.

FIG. 5B illustrates an example LF film network 550, in accordance withone or more embodiments.

FIG. 6 illustrates a side view of a venue 600 which is a traditionaltheater which has been augmented with a LF display system, in accordancewith one or more embodiments.

FIG. 7 illustrates a cross-section of a first venue including a LFdisplay system for displaying a sporting event to viewers at viewinglocations in viewing volumes, in accordance with one or moreembodiments.

FIG. 8 illustrates a venue that also acts as a home theater in theliving room of a viewer, in accordance with one or more embodiments.

FIG. 9 is a flow diagram illustrating a method for displayingholographic content of a sporting event within a LF sporting eventdistribution network.

FIG. 10 is a process flow diagram illustrating a method for displayingholographic content.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION Overview

A light field (LF) display system is implemented in a presentation spacethat acts as a setting for a sporting event. For example, a sportingevent may be a baseball game, a basketball game a hockey game, agymnastics event, a cricket match, a field hockey game, a table tennisgame, a rugby match, a golf game, a track and field meet, a footballgame, a soccer game, a car racing event, a tennis match, a boxing match,a martial arts bout, an Olympics event, and a world championship event.The LF display system comprises a LF display assembly configured topresent holographic content including one or more holographic objectsthat would be visible to one or more viewers in a viewing volume of theLF display system. The holographic objects may be sporting content andmay include a coach, a player, an official, a medical professional, afan, or a participant. A LF display assembly may form a multi-sidedseamless surface over some or all of one or more surfaces (e.g., acourt) in the presentation space. The LF display system can presentholographic content to viewers in the presentation space. A viewergenerally attends a sporting event at the presentation space, but may beany person in a location that can view the holographic content in thepresentation space.

A holographic object of the holographic content may also be augmentedwith other sensory stimuli (e.g., tactile and/or audio). For example,ultrasonic emitters in the LF display system may emit ultrasonicpressure waves that provide a tactile surface for some or all of theholographic object. Holographic content may include additional visualcontent (i.e., 2D or 3D visual content). The coordination of emitters toensure that a cohesive experience is enabled is part of the system inmulti-emitter implementations (i.e., holographic objects providing thecorrect haptic feel and sensory stimuli at any given point in time.)

In some embodiments, the LF display system includes a plurality of LFdisplay modules that form a sporting display area (e.g., a court, afield, a table-top, etc.) in the presentation space. The LF displaymodules forming the sporting display area may be configured to projectholographic content of a sporting event to viewers in the viewing volumeof the LF display within the presentation space. In this disclosure, itis assumed that a ‘viewer in the presentation space’ refers to a viewerin one of the viewing volumes of the LF display system within thepresentation space. Viewing volumes are described in greater detailbelow. In this manner, a viewer in the presentation space can perceive aholographic sporting event on the sporting display area. For example,the LF display system may display two combatants in a martial arts bout,two teams playing a basketball game, two players in a tennis match, orany other content associated with a sporting event. In some embodiments,the LF display system may create holographic content for display toviewers in the presentation space. For example, the LF display systemmay create a cheering section in the presentation space that cheers forone of the teams participating in the presented sporting event. In anembodiment, the presentation space reproduces a portion of a sportingarena. For example, the LF display within the presentation space maygenerate sporting content representing a portion of a stadium or arena(e.g. a hockey ice rink and a portion of the audience), a field, acourt, a gymnasium, or a portion of an amphitheater. In an embodiment,the presentation space may be located in an arena itself, or may belocated at a bar, a home entertainment system, a table-top presentationsystem, or a public presentation system.

In some embodiments, the LF display system may include elements thatenable the system to simultaneously emit at least one type of energy,and, simultaneously, absorb at least one type of energy (for creatingholographic content). For example, a LF display system can emit bothholographic objects for viewing as well as ultrasonic waves for hapticperception. Simultaneously, the LF display system can absorb bothimaging information for tracking of viewers and other scene analysis andultrasonic waves to detect touch response by the users. As an example,such a system may project a holographic ball that is thrown into theaudience in the presentation space. When the holographic ball isvirtually “touched” by a viewer, the LF display system gives the viewerthe illusion that the ball is in their hands. The display systemcomponents that perform energy sensing of the environment may beintegrated into the display surface via bidirectional energy elementsthat both emit and absorb energy, or they may be dedicated sensors thatare separate from the display surface. For example, the LF displaysystem may include dedicated ultrasonic speakers and image capturedevices.

The LF display system can be part of a LF sporting event network. The LFsporting event network allows LF data to be recorded in one location(e.g., a first arena), encoded, transmitted to a different location(e.g., a second arena), decoded, and displayed as holographic content toviewers in a presentation space in the different location. This allowsviewers in multiple locations to perceive a live-sporting eventoccurring in a different location. In some embodiments, the LF displaysystem includes a network system that manages the digital rights of theholographic content.

Light Field Display System

FIG. 1 is a diagram 100 of a light field (LF) display module 110presenting a holographic object 120, in accordance with one or moreembodiments. The LF display module 110 is part of a light field (LF)display system. The LF display system presents holographic contentincluding at least one holographic object using one or more LF displaymodules. The LF display system can present holographic content to one ormultiple viewers. In some embodiments, the LF display system may alsoaugment the holographic content with other sensory content (e.g., touch,audio, smell, temperature, etc.). For example, as discussed below, theprojection of focused ultrasonic sound waves may generate a mid-airtactile sensation that can simulate a surface of some or all of aholographic object. The LF display system includes one or more LFdisplay modules 110, and is discussed in detail below with regard toFIGS. 2-4 .

The LF display module 110 is a holographic display that presentsholographic objects (e.g., the holographic object 120) to one or moreviewers (e.g., viewer 140). The LF display module 110 includes an energydevice layer (e.g., an emissive electronic display or acousticprojection device) and an energy waveguide layer (e.g., an optical lensarray). Additionally, the LF display module 110 may include an energyrelay layer for combining multiple energy sources or detectors togetherto form a single surface. At a high-level, the energy device layergenerates energy (e.g., holographic content) that is then directed usingthe energy waveguide layer to a region in space in accordance with oneor more four-dimensional (4D) light field functions. The LF displaymodule 110 may also project and/or sense one or more types of energysimultaneously. For example, LF display module 110 may be able toproject a holographic image as well as an ultrasonic tactile surface ina viewing volume, while simultaneously detecting imaging data from theviewing volume. The operation of the LF display module 110 is discussedin more detail below with regard to FIGS. 2-3 .

The LF display module 110 generates holographic objects within aholographic object volume 160 using one or more 4D light field functions(e.g., derived from a plenoptic function). The holographic objects canbe three-dimensional (3D), two-dimensional (2D), or some combinationthereof. Moreover, the holographic objects may be polychromatic (e.g.,full color). The holographic objects may be projected in front of thescreen plane, behind the screen plane, or split by the screen plane. Aholographic object 120 can be presented such that it is perceivedanywhere within the holographic object volume 160. A holographic objectwithin the holographic object volume 160 may appear to a viewer 140 tobe floating in space.

A holographic object volume 160 represents a volume in which holographicobjects may be perceived by a viewer 140. The holographic object volume160 can extend in front of the surface of the display area 150 (i.e.,towards the viewer 140) such that holographic objects can be presentedin front of the plane of the display area 150. Additionally, theholographic object volume 160 can extend behind the surface of thedisplay area 150 (i.e., away from the viewer 140), allowing forholographic objects to be presented as if they are behind the plane ofthe display area 150. In other words, the holographic object volume 160may include all the rays of light that originate (e.g., are projected)from a display area 150 and can converge to create a holographic object.Herein, light rays may converge at a point that is in front of thedisplay surface, at the display surface, or behind the display surface.More simply, the holographic object volume 160 encompasses all of thevolume from which a holographic object may be perceived by a viewer.

A viewing volume 130 is a volume of space from which holographic objects(e.g., holographic object 120) presented within a holographic objectvolume 160 by the LF display system are fully viewable. The holographicobjects may be presented within the holographic object volume 160, andviewed within a viewing volume 130, such that they are indistinguishablefrom actual objects. A holographic object is formed by projecting thesame light rays that would be generated from the surface of the objectwere it physically present.

In some cases, the holographic object volume 160 and the correspondingviewing volume 130 may be relatively small—such that it is designed fora single viewer, as described in detail below with regard to FIGS. 10and 11 . In other embodiments, the LF display modules may be enlargedand/or tiled to create larger holographic object volumes andcorresponding viewing volumes that can accommodate a large range ofviewers (e.g., 1 to thousands), as described in detail below with regardto, e.g., FIGS. 4A-4F, 6A-6B, 7A-7B, 8, and 9A-9B. The LF displaymodules presented in this disclosure may be built so that the fullsurface of the LF display contains holographic imaging optics, with noinactive or dead space, and without any need for bezels. In theseembodiments, the LF display modules may be tiled so that the imagingarea is continuous across the seam between LF display modules, and theconnection points between the tiled modules is virtually undetectableusing the visual acuity of the eye. Notably, in some configurations,some portion of the display surface may not include holographic imagingoptics, although they are not described in detail herein.

The flexible size and/or shape of a viewing volume 130 allows forviewers to be unconstrained within the viewing volume 130. For example,a viewer 140 can move to a different position within a viewing volume130 and see a different view of the holographic object 120 from thecorresponding perspective. To illustrate, referring to FIG. 1 , theviewer 140 is at a first position relative to the holographic object 120such that the holographic object 120 appears to be a head-on view of adolphin. The viewer 140 may move to other locations relative to theholographic object 120 to see different views of the dolphin. Forexample, the viewer 140 may move such that he/she sees a left side ofthe dolphin, a right side of the dolphin, etc., much like if the viewer140 was looking at an actual dolphin and changed his/her relativeposition to the actual dolphin to see different views of the dolphin. Insome embodiments, the holographic object 120 is visible to all viewerswithin the viewing volume 130 that have an unobstructed line (i.e., notblocked by an object/person) of sight to the holographic object 120.These viewers may be unconstrained such that they can move around withinthe viewing volume to see different perspectives of the holographicobject 120. Accordingly, the LF display system may present holographicobjects such that a plurality of unconstrained viewers maysimultaneously see different perspectives of the holographic objects inreal-world space as if the holographic objects were physically present.

In contrast, conventional displays (e.g., stereoscopic, virtual reality,augmented reality, or mixed reality) generally require each viewer towear some sort of external device (e.g., 3-D glasses, a near-eyedisplay, or a head-mounted display) to see content. Additionally and/oralternatively, conventional displays may require that a viewer beconstrained to a particular viewing position (e.g., in a chair that hasfixed location relative to the display). For example, when viewing anobject shown by a stereoscopic display, a viewer always focuses on thedisplay surface, rather than on the object, and the display will alwayspresent just two views of an object that will follow a viewer whoattempts to move around that perceived object, causing distortions inthe perception of that object. With a light field display, however,viewers of a holographic object presented by the LF display system donot need to wear an external device, nor be confined to a particularposition, in order to see the holographic object. The LF display systempresents the holographic object in a manner that is visible to viewersin much the same way a physical object would be visible to the viewers,with no requirement of special eyewear, glasses, or a head-mountedaccessory. Further, the viewer may view holographic content from anylocation within a viewing volume.

Notably, potential locations for holographic objects within theholographic object volume 160 are limited by the size of the volume. Toincrease the size of the holographic object volume 160, a size of adisplay area 150 of the LF display module 110 may be increased, and/ormultiple LF display modules may be tiled together in a manner that formsa seamless display surface. The seamless display surface has aneffective display area that is larger than the display areas of theindividual LF display modules. Some embodiments relating to tiling LFdisplay modules are discussed below with regard to FIGS. 4A-4F, 6A-6B,7A-7B, 8, and 9A-9B. As illustrated in FIG. 1 , the display area 150 is,for example, rectangular resulting in a holographic object volume 160that is a pyramid. In other embodiments, the display area may have someother shape (e.g., hexagonal), which also affects the shape of thecorresponding viewing volume.

Additionally, while the above discussion focuses on presenting theholographic object 120 within a portion of the holographic object volume160 that is between the LF display module 110 and the viewer 140, the LFdisplay module 110 can additionally present content in the holographicobject volume 160 behind the plane of the display area 150. For example,the LF display module 110 may make the display area 150 appear to be asurface of the ocean that the holographic object 120 is jumping out of.And the displayed content may be such that the viewer 140 is able tolook through the displayed surface to see marine life that is under thewater. Moreover, the LF display system can generate content thatseamlessly moves around the holographic object volume 160, includingbehind and in front of the plane of the display area 150.

FIG. 2A illustrates a cross section 200 of a portion of a LF displaymodule 210, in accordance with one or more embodiments. The LF displaymodule 210 may be the LF display module 110. In other embodiments, theLF display module 210 may be another LF display module with a differentdisplay area shape than display area 150. In the illustrated embodiment,the LF display module 210 includes an energy device layer 220, an energyrelay layer 230, and an energy waveguide layer 240. Some embodiments ofthe LF display module 210 have different components than those describedhere. For example, in some embodiments, the LF display module 210 doesnot include the energy relay layer 230. Similarly, the functions can bedistributed among the components in a different manner than is describedhere.

The display system described here presents an emission of energy thatreplicates the energy normally surrounding an object in the real world.Here, emitted energy is directed towards a specific direction from everycoordinate on the display surface. In other words, the variouscoordinates on the display surface act as projection locations foremitted energy. The directed energy from the display surface enablesconvergence of many rays of energy, which, thereby, can createholographic objects. For visible light, for example, the LF display willproject a very large number of light rays from the projection locationsthat may converge at any point in the holographic object volume so theywill appear to come from the surface of a real-world object located inthis region of space from the perspective of a viewer that is locatedfurther away than the object being projected. In this way, the LFdisplay is generating the rays of reflected light that would leave suchan object's surface from the perspective of the viewer. The viewerperspective may change on any given holographic object, and the viewerwill see a different view of that holographic object.

The energy device layer 220 includes one or more electronic displays(e.g., an emissive display such as an OLED) and one or more other energyprojection and/or energy receiving devices as described herein. The oneor more electronic displays are configured to display content inaccordance with display instructions (e.g., from a controller of a LFdisplay system). The one or more electronic displays include a pluralityof pixels, each with an intensity that is individually controlled. Manytypes of commercial displays, such as emissive LED and OLED displays,may be used in the LF display.

The energy device layer 220 may also include one or more acousticprojection devices and/or one or more acoustic receiving devices. Anacoustic projection device generates one or more pressure waves thatcomplement the holographic object 250. The generated pressure waves maybe, e.g., audible, ultrasonic, or some combination thereof. An array ofultrasonic pressure waves may be used for volumetric tactile sensation(e.g., at a surface of the holographic object 250). An audible pressurewave is used for providing audio content (e.g., immersive audio) thatcan complement the holographic object 250. For example, assuming theholographic object 250 is a dolphin, one or more acoustic projectiondevices may be used to (1) generate a tactile surface that is collocatedwith a surface of the dolphin such that viewers may touch theholographic object 250; and (2) provide audio content corresponding tonoises a dolphin makes such as clicks, chirping, or chatter. An acousticreceiving device (e.g., a microphone or microphone array) may beconfigured to monitor ultrasonic and/or audible pressure waves within alocal area of the LF display module 210.

The energy device layer 220 may also include one or more imagingsensors. An imaging sensor may be sensitive to light in a visibleoptical band, and in some cases may be sensitive to light in other bands(e.g., infrared). The imaging sensor may be, e.g., a complementary metaloxide semi-conductor (CMOS) array, a charged coupled device (CCD), anarray of photodetectors, some other sensor that captures light, or somecombination thereof. The LF display system may use data captured by theone or more imaging sensor for position location tracking of viewers.

In some configurations, the energy relay layer 230 relays energy (e.g.,electromagnetic energy, mechanical pressure waves, etc.) between theenergy device layer 220 and the energy waveguide layer 240. The energyrelay layer 230 includes one or more energy relay elements 260. Eachenergy relay element includes a first surface 265 and a second surface270, and it relays energy between the two surfaces. The first surface265 of each energy relay element may be coupled to one or more energydevices (e.g., electronic display or acoustic projection device). Anenergy relay element may be composed of, e.g., glass, carbon, opticalfiber, optical film, plastic, polymer, or some combination thereof.Additionally, in some embodiments, an energy relay element may adjustmagnification (increase or decrease) of energy passing between the firstsurface 265 and the second surface 270. If the relay offersmagnification, then the relay may take the form of an array of bondedtapered relays, called tapers, where the area of one end of the tapermay be substantially larger than the opposite end. The large end of thetapers can be bonded together to form a seamless energy surface 275. Oneadvantage is that space is created on the multiple small ends of eachtaper to accommodate the mechanical envelope of multiple energy sources,such as the bezels of multiple displays. This extra room allows theenergy sources to be placed side-by-side on the small taper side, witheach energy source having their active areas directing energy into thesmall taper surface and relayed to the large seamless energy surface.Another advantage to using tapered relays is that there is nonon-imaging dead space on the combined seamless energy surface formed bythe large end of the tapers. No border or bezel exists, and so theseamless energy surfaces can then be tiled together to form a largersurface with virtually no seams according to the visual acuity of theeye.

The second surfaces of adjacent energy relay elements come together toform an energy surface 275. In some embodiments, a separation betweenedges of adjacent energy relay elements is less than a minimumperceptible contour as defined by a visual acuity of a human eye having,for example, 20/40 vision, such that the energy surface 275 iseffectively seamless from the perspective of a viewer 280 within aviewing volume 285.

In some embodiments, the second surfaces of adjacent energy relayelements are fused together with processing steps that may include oneor more of pressure, heat, and a chemical reaction, in such a way noseam exists between them. And still in other embodiments, an array ofenergy relay elements is formed by molding one side of a continuousblock of relay material into an array of small taper ends, eachconfigured to transport energy from an energy device attached to thesmall tapered end into a single combined surface with a larger areawhich is never subdivided.

In some embodiments, one or more of the energy relay elements exhibitenergy localization, where the energy transport efficiency in thelongitudinal direction substantially normal to the surfaces 265 and 270is much higher than the transport efficiency in the perpendiculartransverse plane, and where the energy density is highly localized inthis transverse plane as the energy wave propagates between surface 265and surface 270. This localization of energy allows an energydistribution, such as an image, to be efficiency relayed between thesesurfaces without any significant loss in resolution.

The energy waveguide layer 240 directs energy from a location (e.g., acoordinate) on the energy surface 275 into a specific energy propagationpath outward from the display surface into the holographic viewingvolume 285 using waveguide elements in the energy waveguide layer 240.The energy propagation path is defined by two angular dimensionsdetermined at least by the energy surface coordinate location relativeto the waveguide. The waveguide is associated with a spatial 2Dcoordinate. Together, these four coordinates form a four-dimensional(4D) energy field. As an example, for electromagnetic energy, thewaveguide elements in the energy waveguide layer 240 direct light frompositions on the seamless energy surface 275 along different propagationdirections through the viewing volume 285. In various examples, thelight is directed in accordance with a 4D light field function to formthe holographic object 250 within the holographic object volume 255.

Each waveguide element in the energy waveguide layer 240 may be, forexample, a lenslet composed of one or more elements. In someconfigurations, the lenslet may be a positive lens. The positive lensmay have a surface profile that is spherical, aspherical, or freeform.Additionally, in some embodiments, some or all of the waveguide elementsmay include one or more additional optical components. An additionaloptical component may be, e.g., an energy-inhibiting structure such as abaffle, a positive lens, a negative lens, a spherical lens, anaspherical lens, a freeform lens, a liquid crystal lens, a liquid lens,a refractive element, a diffractive element, or some combinationthereof. In some embodiments, the lenslet and/or at least one of theadditional optical components is able to dynamically adjust its opticalpower. For example, the lenslet may be a liquid crystal lens or a liquidlens. Dynamic adjustment of a surface profile the lenslet and/or atleast one additional optical component may provide additionaldirectional control of light projected from a waveguide element.

In the illustrated example, the holographic object volume 255 of the LFdisplay has boundaries formed by light ray 256 and light ray 257, butcould be formed by other rays. The holographic object volume 255 is acontinuous volume that extends both in front (i.e., towards the viewer280) of the energy waveguide layer 240 and behind it (i.e., away fromthe viewer 280). In the illustrated example, ray 256 and ray 257 areprojected from opposite edges of the LF display module 210 at thehighest angle relative to the normal to the display surface 277 that maybe perceived by a user, but these could be other projected rays. Therays define the field-of-view of the display, and, thus, define theboundaries for the holographic viewing volume 285. In some cases, therays define a holographic viewing volume where the full display can beobserved without vignetting (e.g., an ideal viewing volume). As thefield of view of the display increases, the convergence point of ray 256and ray 257 will be closer to the display. Thus, a display having alarger field of view allows a viewer 280 to see the full display at acloser viewing distance. Additionally, ray 256 and 257 may form an idealholographic object volume. Holographic objects presented in an idealholographic object volume can be seen anywhere in the viewing volume285.

In some examples, holographic objects may be presented to only a portionof the viewing volume 285. In other words, holographic object volumesmay be divided into any number of viewing sub-volumes (e.g., viewingsub-volume 290). Additionally, holographic objects can be projectedoutside of the holographic object volume 255. For example, holographicobject 251 is presented outside of holographic object volume 255.Because the holographic object 251 is presented outside of theholographic object volume 255 it cannot be viewed from every location inthe viewing volume 285. For example, holographic object 251 may bevisible from a location in viewing sub-volume 290, but not visible fromthe location of the viewer 280.

For example, we turn to FIG. 2B to illustrate viewing holographiccontent from different viewing sub-volumes. FIG. 2B illustrates a crosssection 200 of a portion of a LF display module, in accordance with oneor more embodiments. The cross-section of FIG. 2B is the same as thecross-section of FIG. 2A. However, FIG. 2B illustrates a different setof light rays projected from the LF display module 210. Ray 256 and ray257 still form a holographic object volume 255 and a viewing volume 285.However, as shown, rays projected from the top of the LF display module210 and the bottom of the LF display module 210 overlap to form variousviewing sub-volumes (e.g., view sub-volumes 290A, 290B, 290C, and 290D)within the viewing volume 285. A viewer in the first viewing sub-volume(e.g., 290A) may be able to perceive holographic content presented inthe holographic object volume 255 that viewers in the other viewingsub-volumes (e.g., 290B, 290C, and 290D) are unable to perceive.

More simply, as illustrated in FIG. 2A, holographic object volume 255 isa volume in which holographic objects may be presented by LF displaysystem such that they may be perceived by viewers (e.g., viewer 280) inviewing volume 285. In this way, the viewing volume 285 is an example ofan ideal viewing volume, while the holographic object volume 255 is anexample of an ideal object volume. However, in various configurations,viewers may perceive holographic objects presented by LF display system200 in other example holographic object volumes. More generally, an“eye-line guideline” applies when viewing holographic content projectedfrom an LF display module. The eye-line guideline asserts that the lineformed by a viewer's eye position and a holographic object being viewedmust intersect a LF display surface.

When viewing holographic content presented by the LF display module 210,each eye of the viewer 280 sees a different perspective of theholographic object 250 because the holographic content is presentedaccording to a 4D light field function. Moreover, as the viewer 280moves within the viewing volume 285 he/she would also see differentperspectives of the holographic object 250 as would other viewers withinthe viewing volume 285. As will be appreciated by one of ordinary skillin the art, a 4D light field function is well known in the art and willnot be elaborated further herein.

As described in more detail herein, in some embodiments, the LF displaycan project more than one type of energy. For example, the LF displaymay project two types of energy, such as, for example, mechanical energyand electromagnetic energy. In this configuration, energy relay layer230 may include two separate energy relays which are interleavedtogether at the energy surface 275, but are separated such that theenergy is relayed to two different energy device layers 220. Here, onerelay may be configured to transport electromagnetic energy, whileanother relay may be configured to transport mechanical energy. In someembodiments, the mechanical energy may be projected from locationsbetween the electromagnetic waveguide elements on the energy waveguidelayer 240, helping form structures that inhibit light from beingtransported from one electromagnetic waveguide element to another. Insome embodiments, the energy waveguide layer 240 may also includewaveguide elements that transport focused ultrasound along specificpropagation paths in accordance with display instructions from acontroller.

Note that in alternate embodiments (not shown), the LF display module210 does not include the energy relay layer 230. In this case, theenergy surface 275 is an emission surface formed using one or moreadjacent electronic displays within the energy device layer 220. And insome embodiments, with no energy relay layer, a separation between edgesof adjacent electronic displays is less than a minimum perceptiblecontour as defined by a visual acuity of a human eye having 20/40vision, such that the energy surface is effectively seamless from theperspective of the viewer 280 within the viewing volume 285.

LF Display Modules

FIG. 3A is a perspective view of a LF display module 300A, in accordancewith one or more embodiments. The LF display module 300A may be the LFdisplay module 110 and/or the LF display module 210. In otherembodiments, the LF display module 300A may be some other LF displaymodule. In the illustrated embodiment, the LF display module 300Aincludes an energy device layer 310, and energy relay layer 320, and anenergy waveguide layer 330. The LF display module 300A is configured topresent holographic content from a display surface 365 as describedherein. For convenience, the display surface 365 is illustrated as adashed outline on the frame 390 of the LF display module 300A, but is,more accurately, the surface directly in front of waveguide elementsbounded by the inner rim of the frame 390. The display surface 365includes a plurality of projection locations from which energy can beprojected. Some embodiments of the LF display module 300A have differentcomponents than those described here. For example, in some embodiments,the LF display module 300A does not include the energy relay layer 320.Similarly, the functions can be distributed among the components in adifferent manner than is described here.

The energy device layer 310 is an embodiment of the energy device layer220. The energy device layer 310 includes four energy devices 340 (threeare visible in the figure). The energy devices 340 may all be the sametype (e.g., all electronic displays), or may include one or moredifferent types (e.g., includes electronic displays and at least oneacoustic energy device).

The energy relay layer 320 is an embodiment of the energy relay layer230. The energy relay layer 320 includes four energy relay devices 350(three are visible in the figure). The energy relay devices 350 may allrelay the same type of energy (e.g., light), or may relay one or moredifferent types (e.g., light and sound). Each of the relay devices 350includes a first surface and a second surface, the second surface of theenergy relay devices 350 being arranged to form a singular seamlessenergy surface 360. In the illustrated embodiment, each of the energyrelay devices 350 are tapered such that the first surface has a smallersurface area than the second surface, which allows accommodation for themechanical envelopes of the energy devices 340 on the small end of thetapers. This also allows the seamless energy surface to be borderless,since the entire area can project energy. This means that this seamlessenergy surface can be tiled by placing multiple instances of LF displaymodule 300A together, without dead space or bezels, so that the entirecombined surface is seamless. In other embodiments, the first surfaceand the second surface have the same surface area.

The energy waveguide layer 330 is an embodiment of the energy waveguidelayer 240. The energy waveguide layer 330 includes a plurality ofwaveguide elements 370. As discussed above with respect to FIG. 2 , theenergy waveguide layer 330 is configured to direct energy from theseamless energy surface 360 along specific propagation paths inaccordance with a 4D light field function to form a holographic object.Note that in the illustrated embodiment the energy waveguide layer 330is bounded by a frame 390. In other embodiments, there is no frame 390and/or a thickness of the frame 390 is reduced. Removal or reduction ofthickness of the frame 390 can facilitate tiling the LF display module300A with additional LF display modules.

Note that in the illustrated embodiment, the seamless energy surface 360and the energy waveguide layer 330 are planar. In alternate embodiments,not shown, the seamless energy surface 360 and the energy waveguidelayer 330 may be curved in one or more dimensions.

The LF display module 300A can be configured with additional energysources that reside on the surface of the seamless energy surface, andallow the projection of an energy field in additional to the lightfield. In one embodiment, an acoustic energy field may be projected fromelectrostatic speakers (not illustrated) mounted at any number oflocations on the seamless energy surface 360. Further, the electrostaticspeakers of the LF display module 300A are positioned within the lightfield display module 300A such that the dual-energy surfacesimultaneously projects sound fields and holographic content. Forexample, the electrostatic speakers may be formed with one or morediaphragm elements that are transmissive to some wavelengths ofelectromagnetic energy, and driven with one or more conductive elements(e.g., planes which sandwich the one or more diaphragm elements). Theelectrostatic speakers may be mounted on to the seamless energy surface360, so that the diaphragm elements cover some of the waveguideelements. The conductive electrodes of the speakers may be co-locatedwith structures designed to inhibit light transmission betweenelectromagnetic waveguides, and/or located at positions betweenelectromagnetic waveguide elements (e.g., frame 390). In variousconfigurations, the speakers can project an audible sound and/or manysources of focused ultrasonic energy that produces a haptic surface.

In some configurations an energy device 340 may sense energy. Forexample, an energy device may be a microphone, a light sensor, anacoustic transducer, etc. As such, the energy relay devices may alsorelay energy from the seamless energy surface 360 to the energy devicelayer 310. That is, the seamless energy surface 360 of the LF displaymodule forms a bidirectional energy surface when the energy devices andenergy relay devices 340 are configured to simultaneously emit and senseenergy (e.g., emit light fields and sense sound).

More broadly, an energy device 340 of a LF display module 340 can beeither an energy source or an energy sensor. The LF display module 300Acan include various types of energy devices that act as energy sourcesand/or energy sensors to facilitate the projection of high qualityholographic content to a user. Other sources and/or sensors may includethermal sensors or sources, infrared sensors or sources, image sensorsor sources, mechanical energy transducers that generate acoustic energy,feedback sources, etc. Many other sensors or sources are possible.Further, the LF display modules can be tiled such that the LF displaymodule can form an assembly that projects and senses multiple types ofenergy from a large aggregate seamless energy surface

In various embodiments of LF display module 300A, the seamless energysurface 360 can have various surface portions where each surface portionis configured to project and/or emit specific types of energy. Forexample, when the seamless energy surface is a dual-energy surface, theseamless energy surface 360 includes one or more surface portions thatproject electromagnetic energy, and one or more other surface portionsthat project ultrasonic energy. The surface portions that projectultrasonic energy may be located on the seamless energy surface 360between electromagnetic waveguide elements, and/or co-located withstructures designed to inhibit light transmission betweenelectromagnetic waveguide elements. In an example where the seamlessenergy surface is a bidirectional energy surface, the energy relay layer320 may include two types of energy relay devices interleaved at theseamless energy surface 360. In various embodiments, the seamless energysurface 360 may be configured such that portions of the surface underany particular waveguide element 370 are all energy sources, all energysensors, or a mix of energy sources and energy sensors.

FIG. 3B is a cross-sectional view of a LF display module 300B whichincludes interleaved energy relay devices, in accordance with one ormore embodiments. Energy relay device 350A transports energy between theenergy relay first surface 345A connected to energy device 340A, and theseamless energy surface 360. Energy relay 350B transports energy betweenthe energy relay first surface 345B connected to energy device 340B, andthe seamless energy surface 360. Both relay devices are interleaved atinterleaved energy relay device 352, which is connected to the seamlessenergy surface 360. In this configuration, surface 360 containsinterleaved energy locations of both energy devices 340A and 340B, whichmay be energy sources or energy sensors. Accordingly, the LF displaymodule 300B may be configured as either a dual energy projection devicefor projecting more than one type of energy, or as a bidirectionalenergy device for simultaneously projecting one type of energy andsensing another type of energy. The LF display module 300B may be the LFdisplay module 110 and/or the LF display module 210. In otherembodiments, the LF display module 300B may be some other LF displaymodule.

The LF display module 300B includes many components similarly configuredto those of LF display module 300A in FIG. 3A. For example, in theillustrated embodiment, the LF display module 300B includes an energydevice layer 310, energy relay layer 320, a seamless energy surface 360,and an energy waveguide layer 330 including at least the samefunctionality of those described in regard to FIG. 3A. Additionally, theLF display module 300B may present and/or receive energy from thedisplay surface 365. Notably, the components of the LF display module300B are alternatively connected and/or oriented than those of the LFdisplay module 300A in FIG. 3A. Some embodiments of the LF displaymodule 300B have different components than those described here.Similarly, the functions can be distributed among the components in adifferent manner than is described here. FIG. 3B illustrates the designof a single LF display module 300B that may be tiled to produce a dualenergy projection surface or a bidirectional energy surface with alarger area.

In an embodiment, the LF display module 300B is a LF display module of abidirectional LF display system. A bidirectional LF display system maysimultaneously project energy and sense energy from the display surface365. The seamless energy surface 360 contains both energy projecting andenergy sensing locations that are closely interleaved on the seamlessenergy surface 360. Therefore, in the example of FIG. 3B, the energyrelay layer 320 is configured in a different manner than the energyrelay layer of FIG. 3A. For convenience, the energy relay layer of LFdisplay module 300B will be referred to herein as the “interleavedenergy relay layer.”

The interleaved energy relay layer 320 includes two legs: a first energyrelay device 350A and a second energy relay device 350B. Each of thelegs are illustrated as a lightly shaded area in FIG. 3B. Each of thelegs may be made of a flexible relay material, and formed with asufficient length to use with energy devices of various sizes andshapes. In some regions of the interleaved energy relay layer, the twolegs are tightly interleaved together as they approach the seamlessenergy surface 360. In the illustrated example, the interleaved energyrelay devices 352 are illustrated as a darkly shaded area.

While interleaved at the seamless energy surface 360, the energy relaydevices are configured to relay energy to/from different energy devices.The energy devices are at energy device layer 310. As illustrated,energy device 340A is connected to energy relay device 350A and energydevice 340B is connected to energy relay device 350B. In variousembodiments, each energy device may be an energy source or energysensor.

An energy waveguide layer 330 includes waveguide elements 370 to steerenergy waves from the seamless energy surface 360 along projected pathstowards a series of convergence points. In this example, a holographicobject 380 is formed at the series of convergence points. Notably, asillustrated, the convergence of energy at the holographic object 380occurs on the viewer side (i.e., the front side), of the display surface365. However, in other examples, the convergence of energy may beanywhere in the holographic object volume, which extends both in frontof the display surface 365 and behind the display surface 365. Thewaveguide elements 370 can simultaneously steer incoming energy to anenergy device (e.g., an energy sensor), as described below.

In one example embodiment of LF display module 300B, an emissive displayis used as an energy source (e.g., energy device 340A) and an imagingsensor is used as an energy sensor (e.g., energy device 340B). In thismanner, the LF display module 300B can simultaneously projectholographic content and detect light from the volume in front of thedisplay surface 365. In this manner, this embodiment of the LF displaymodule 300B functions as both a LF display and an LF sensor.

In an embodiment, the LF display module 300B is configured tosimultaneously project a light field from projection locations on thedisplay surface to the front of the display surface and capture a lightfield from front of the display surface at the projection locations. Inthis embodiment, the energy relay device 350A connects a first set oflocations at the seamless energy surface 360 positioned under thewaveguide elements 370 to an energy device 340A. In an example, energydevice 340A is an emissive display having an array of source pixels. Theenergy relay device 340B connects a second set of locations at theseamless energy surface 360 positioned under waveguide elements 370 toan energy device 340B. In an example, the energy device 340B is animaging sensor having an array of sensor pixels. The LF display module300B may be configured such that the locations at the seamless energysurface 365 that are under a particular waveguide element 370 are allemissive display locations, all imaging sensor locations, or somecombination of these locations. In other embodiments, the bidirectionalenergy surface can project and receive various other forms of energy.

In another example embodiment of the LF display module 300B, the LFdisplay module is configured to project two different types of energy.For example, in an embodiment, energy device 340A is an emissive displayconfigured to emit electromagnetic energy and energy device 340B is anultrasonic transducer configured to emit mechanical energy. As such,both light and sound can be projected from various locations at theseamless energy surface 360. In this configuration, energy relay device350A connects the energy device 340A to the seamless energy surface 360and relays the electromagnetic energy. The energy relay device isconfigured to have properties (e.g. varying refractive index) which makeit efficient for transporting electromagnetic energy. Energy relaydevice 350B connects the energy device 340B to the seamless energysurface 360 and relays mechanical energy. Energy relay device 350B isconfigured to have properties for efficient transport of ultrasoundenergy (e.g. distribution of materials with different acousticimpedance). In some embodiments, the mechanical energy may be projectedfrom locations between the waveguide elements 370 on the energywaveguide layer 330. The locations that project mechanical energy mayform structures that serve to inhibit light from being transported fromone electromagnetic waveguide element to another. In one example, aspatially separated array of locations that project ultrasonicmechanical energy can be configured to create three-dimensional hapticshapes and surfaces in mid-air. The surfaces may coincide with projectedholographic objects (e.g., holographic object 380). In some examples,phase delays and amplitude variations across the array can assist increating the haptic shapes.

In various embodiments, the LF display module 300B with interleavedenergy relay devices may include multiple energy device layers with eachenergy device layer including a specific type of energy device. In theseexamples, the energy relay layers are configured to relay theappropriate type of energy between the seamless energy surface 360 andthe energy device layer 310.

Tiled LF Display Modules

FIG. 4A is a perspective view of a portion of LF display system 400 thatis tiled in two dimensions to form a single-sided seamless surfaceenvironment, in accordance with one or more embodiments. The LF displaysystem 400 includes a plurality of LF display modules that are tiled toform an array 410. More explicitly, each of the small squares in thearray 410 represents a tiled LF display module 412. The LF displaymodule 412 may be the same as LF display module 300A or 300B. The array410 may cover, for example, some or all of a surface (e.g., a wall) of aroom. The LF array may cover other surfaces, such as, for example, atable top, a billboard, a rotunda, etc.

The array 410 may project one or more holographic objects. For example,in the illustrated embodiment, the array 410 projects a holographicobject 420 and a holographic object 422. Tiling of the LF displaymodules 412 allows for a much larger viewing volume as well as allowsfor objects to be projected out farther distances from the array 410.For example, in the illustrated embodiment, the viewing volume is,approximately, the entire area in front of and behind the array 410rather than a localized volume in front of (and behind) a LF displaymodule 412.

In some embodiments, the LF display system 400 presents the holographicobject 420 to a viewer 430 and a viewer 434. The viewer 430 and theviewer 434 receive different perspectives of the holographic object 420.For example, the viewer 430 is presented with a direct view of theholographic object 420, whereas the viewer 434 is presented with a moreoblique view of the holographic object 420. As the viewer 430 and/or theviewer 434 move, they are presented with different perspectives of theholographic object 420. This allows a viewer to visually interact with aholographic object by moving relative to the holographic object. Forexample, as the viewer 430 walks around a holographic object 420, theviewer 430 sees different sides of the holographic object 420 as long asthe holographic object 420 remains in the holographic object volume ofthe array 410. Accordingly, the viewer 430 and the viewer 434 maysimultaneously see the holographic object 420 in real-world space as ifit is truly there. Additionally, the viewer 430 and the viewer 434 donot need to wear an external device in order to see the holographicobject 420, as the holographic object 420 is visible to viewers in muchthe same way a physical object would be visible. Additionally, here, theholographic object 422 is illustrated behind the array because theviewing volume of the array extends behind the surface of the array. Inthis manner, the holographic object 422 may be presented to the viewer430 and/or viewer 434.

In some embodiments, the LF display system 400 may include a trackingsystem that tracks positions of the viewer 430 and the viewer 434. Insome embodiments, the tracked position is the position of a viewer. Inother embodiments, the tracked position is that of the eyes of a viewer.The position tracking of the eye is different from gaze tracking whichtracks where an eye is looking (e.g., uses orientation to determine gazelocation). The eyes of the viewer 430 and the eyes of the viewer 434 arein different locations.

In various configurations, the LF display system 400 may include one ormore tracking systems. For example, in the illustrated embodiment ofFIG. 4A, LF display system includes a tracking system 440 that isexternal to the array 410. Here, the tracking system may be a camerasystem coupled to the array 410. External tracking systems are describedin more detail in regard to FIG. 5A. In other example embodiments, thetracking system may be incorporated into the array 410 as describedherein. For example, an energy device (e.g., energy device 340) of oneor more LF display modules 412 containing a bidirectional energy surfaceincluded in the array 410 may be configured to capture images of viewersin front of the array 410. In whichever case, the tracking system(s) ofthe LF display system 400 determines tracking information about theviewers (e.g., viewer 430 and/or viewer 434) viewing holographic contentpresented by the array 410.

Tracking information describes a position in space (e.g., relative tothe tracking system) for the position of a viewer, or a position of aportion of a viewer (e.g. one or both eyes of a viewer, or theextremities of a viewer). A tracking system may use any number of depthdetermination techniques to determine tracking information. The depthdetermination techniques may include, e.g., structured light, time offlight, stereo imaging, some other depth determination technique, orsome combination thereof. The tracking system may include varioussystems configured to determine tracking information. For example, thetracking system may include one or more infrared sources (e.g.,structured light sources), one or more imaging sensors that can captureimages in the infrared (e.g., red-blue-green-infrared camera), and aprocessor executing tracking algorithms. The tracking system may use thedepth estimation techniques to determine positions of viewers. In someembodiments, the LF display system 400 generates holographic objectsbased on tracked positions, motions, or gestures of the viewer 430and/or the viewer 434 as described herein. For example, the LF displaysystem 400 may generate a holographic object responsive to a viewercoming within a threshold distance of the array 410 and/or a particularposition.

The LF display system 400 may present one or more holographic objectsthat are customized to each viewer based in part on the trackinginformation. For example, the viewer 430 may be presented with theholographic object 420, but not the holographic object 422. Similarly,the viewer 434 may be presented with the holographic object 422, but notthe holographic object 420. For example, the LF display system 400tracks a position of each of the viewer 430 and the viewer 434. The LFdisplay system 400 determines a perspective of a holographic object thatshould be visible to a viewer based on their position relative to wherethe holographic object is to be presented. The LF display system 400selectively projects light from specific pixels that correspond to thedetermined perspective. Accordingly, the viewer 434 and the viewer 430can simultaneously have experiences that are, potentially, completelydifferent. In other words, the LF display system 400 may presentholographic content to viewing sub-volumes of the viewing volume (i.e.,similar to the viewing sub-volumes 290A, 290B, 290C, and 290D shown inFIG. 2B). For example, as illustrated, because the LF display system 400can track the position of the viewer 430, the LF display system 400 maypresent space content (e.g., holographic object 420) to a viewingsub-volume surrounding the viewer 430 and safari content (e.g.,holographic object 422) to a viewing sub-volume surrounding the viewer434. In contrast, conventional systems would have to use individualheadsets to provide a similar experience.

In some embodiments the LF display system 400 may include one or moresensory feedback systems. The sensory feedback systems provide othersensory stimuli (e.g., tactile, audio, or smell) that augment theholographic objects 420 and 422. For example, in the illustratedembodiment of FIG. 4A, the LF display system 400 includes a sensoryfeedback system 442 external to the array 410. In one example, thesensory feedback system 442 may be an electrostatic speaker coupled tothe array 410. External sensory feedback systems are described in moredetail in regard to FIG. 5A. In other example embodiments, the sensoryfeedback system may be incorporated into the array 410 as describedherein. For example, an energy device (e.g., energy device 340A in FIG.3B) of a LF display module 412 included in the array 410 may beconfigured to project ultrasonic energy to viewers in front of the arrayand/or receive imaging information from viewers in front of the array.In whichever case, the sensory feedback system presents and/or receivessensory content to/from the viewers (e.g., viewer 430 and/or viewer 434)viewing holographic content (e.g., holographic object 420 and/orholographic objected 422) presented by the array 410.

The LF display system 400 may include a sensory feedback system 442 thatincludes one or more acoustic projection devices external to the array.Alternatively or additionally, the LF display system 400 may include oneor more acoustic projection devices integrated into the array 410 asdescribed herein. The acoustic projection devices may consist of anarray of ultrasonic sources configured to project a volumetric tactilesurface. In some embodiments, the tactile surface may be coincident witha holographic object (e.g., at a surface of the holographic object 420)for one or more surfaces of a holographic object if a portion of aviewer gets within a threshold distance of the one or more surfaces. Thevolumetric tactile sensation may allow the user to touch and feelsurfaces of the holographic object. The plurality of acoustic projectiondevices may also project an audible pressure wave that provides audiocontent (e.g., immersive audio) to viewers. Accordingly, the ultrasonicpressure waves and/or the audible pressure waves can act to complement aholographic object.

In various embodiments, the LF display system 400 may provide othersensory stimuli based in part on a tracked position of a viewer. Forexample, the holographic object 422 illustrated in FIG. 4A is a lion,and the LF display system 400 may have the holographic object 422 roarboth visually (i.e., the holographic object 422 appears to roar) andaudibly (i.e., one or more acoustic projection devices project apressure wave that the viewer 430 perceives as a lion's roar emanatingfrom the holographic object 422.

Note that, in the illustrated configuration, the holographic viewingvolume may be limited in a manner similar to the viewing volume 285 ofthe LF display system 200 in FIG. 2 . This can limit the amount ofperceived immersion that a viewer will experience with a single walldisplay unit. One way to address this is to use multiple LF displaymodules that are tiled along multiple sides as described below withrespect to FIG. 4B-4F.

FIG. 4B is a perspective view of a portion of a LF display system 402 ina multi-sided seamless surface environment, in accordance with one ormore embodiments. The LF display system 402 is substantially similar tothe LF display system 400 except that the plurality of LF displaymodules are tiled to create a multi-sided seamless surface environment.More specifically, the LF display modules are tiled to form an arraythat is a six-sided aggregated seamless surface environment. In FIG. 4B,the plurality of LF display modules cover all the walls, the ceiling,and the floor of a room. In other embodiments, the plurality of LFdisplay modules may cover some, but not all of a wall, a floor, aceiling, or some combination thereof. In other embodiments, a pluralityof LF display modules are tiled to form some other aggregated seamlesssurface. For example, the walls may be curved such that a cylindricalaggregated energy environment is formed. Moreover, as described belowwith regard to FIGS. 6-9 , in some embodiments, the LF display modulesmay be tiled to form a surface in a presentation space (e.g., walls,etc.).

The LF display system 402 may project one or more holographic objects.For example, in the illustrated embodiment the LF display system 402projects the holographic object 420 into an area enclosed by thesix-sided aggregated seamless surface environment. In this example, theviewing volume of the LF display system is also contained within thesix-sided aggregated seamless surface environment. Note that, in theillustrated configuration, the viewer 434 may be positioned between theholographic object 420 and a LF display module 414 that is projectingenergy (e.g., light and/or pressure waves) that is used to form theholographic object 420. Accordingly, the positioning of the viewer 434may prevent the viewer 430 from perceiving the holographic object 420formed from energy from the LF display module 414. However, in theillustrated configuration there is at least one other LF display module,e.g., a LF display module 416, that is unobstructed (e.g., by the viewer434) and can project energy to form the holographic object 420 and beobserved by viewer 430. In this manner, occlusion by viewers in thespace can cause some portion of the holographic projections todisappear, but the effect is much less than if only one side of thevolume was populated with holographic display panels. Holographic object422 is illustrated “outside” the walls of the six-sided aggregatedseamless surface environment because the holographic object volumeextends behind the aggregated surface. Thus, the viewer 430 and/or theviewer 434 can perceive the holographic object 422 as “outside” of theenclosed six-sided environment which they can move throughout.

As described above in reference to FIG. 4A, in some embodiments, the LFdisplay system 402 actively tracks positions of viewers and maydynamically instruct different LF display modules to present holographiccontent based on the tracked positions. Accordingly, a multi-sidedconfiguration can provide a more robust environment (e.g., relative toFIG. 4A) for providing holographic objects where unconstrained viewersare free to move throughout the area enclosed by the multi-sidedseamless surface environment.

Notably, various LF display systems may have different configurations.Further, each configuration may have a particular orientation ofsurfaces that, in aggregate, form a seamless display surface (“aggregatesurface”). That is, the LF display modules of a LF display system can betiled to form a variety of aggregate surfaces. For example, in FIG. 4B,the LF display system 402 includes LF display modules tiled to form asix-sided aggregate surface that approximates the walls of a room. Insome other examples, an aggregate surface may only occur on a portion ofa surface (e.g., half of a wall) rather than a whole surface (e.g., anentire wall). Some examples are described herein.

In some configurations, the aggregate surface of a LF display system mayinclude an aggregate surface configured to project energy towards alocalized viewing volume. Projecting energy to a localized viewingvolume allows for a higher quality viewing experience by, for example,increasing the density of projected energy in a specific viewing volume,increasing the FOV for the viewers in that volume, and bringing theviewing volume closer to the display surface.

For example, FIG. 4C illustrates top down view of a LF display system450A with an aggregate surface in a “winged” configuration. In thisexample, the LF display system 450A is located in a room with a frontwall 452, a rear wall 454, a first sidewall 456, a second sidewall 458,a ceiling (not shown), and a floor (not shown). The first sidewall 456,the second sidewall 458, the rear wall 454, floor, and the ceiling areall orthogonal. The LF display system 450A includes LF display modulestiled to form an aggregate surface 460 covering the front wall. Thefront wall 452, and thus the aggregate surface 460, includes threeportions: (i) a first portion 462 approximately parallel with the rearwall 454 (i.e., a central surface), (ii) a second portion 464 connectingthe first portion 462 to the first sidewall 456 and placed at an angleto project energy towards the center of the room (i.e., a first sidesurface), and (iii) a third portion 466 connecting the first portion 462to the second sidewall 458 and placed at an angle to project energytowards the center of the room (i.e., a second side surface). The firstportion is a vertical plane in the room and has a horizontal and avertical axis. The second and third portions are angled towards thecenter of the room along the horizontal axis.

In this example, the viewing volume 468A of the LF display system 450Ais in the center of the room and partially surrounded by the threeportions of the aggregate surface 460. An aggregate surface that atleast partially surrounds a viewer (“surrounding surface”) increases theimmersive experience of the viewers.

To illustrate, consider, for example, an aggregate surface with only acentral surface. Referring to FIG. 2A, the rays that are projected fromeither end of the display surface create an ideal holographic volume andideal viewing volumes as described above. Now consider, for example, ifthe central surface included two side surfaces angled towards theviewer. In this case, ray 256 and ray 257 would be projected at agreater angle from a normal of the central surface. Thus, the field ofview of the viewing volume would increase. Similarly, the holographicviewing volume would be nearer the display surface. Additionally,because the two second and third portions tilted nearer the viewingvolume, the holographic objects that are projected at a fixed distancefrom the display surface are closer to that viewing volume.

To simplify, a display surface with only a central surface has a planarfield of view, a planar threshold separation between the (central)display surface and the viewing volume, and a planar proximity between aholographic object and the viewing volume. Adding one or more sidesurfaces angled towards the viewer increases the field of view relativeto the planar field of view, decreases the separation between thedisplay surface and the viewing volume relative to the planarseparation, and increases the proximity between the display surface anda holographic object relative to the planar proximity. Further anglingthe side surfaces towards the viewer further increases the field ofview, decreases the separation, and increases the proximity. In otherwords, the angled placement of the side surfaces increases the immersiveexperience for viewers.

Additionally, as described below in regards to FIG. 6 , deflectionoptics may be used to optimize the size and position of the viewingvolume for LF display parameters (e.g., dimensions and FOV).

Returning to FIG. 4D, in a similar example, FIG. 4D illustrates a sideview of a LF display system 450B with an aggregate surface in a “sloped”configuration. In this example, the LF display system 450B is located ina room with a front wall 452, a rear wall 454, a first sidewall (notshown), a second sidewall (not shown), a ceiling 472, and a floor 474.The first sidewall, the second sidewall, the rear wall 454, floor 474,and the ceiling 472 are all orthogonal. The LF display system 450Bincludes LF display modules tiled to form an aggregate surface 460covering the front wall. The front wall 452, and thus the aggregatesurface 460, includes three portions: (i) a first portion 462approximately parallel with the rear wall 454 (i.e., a central surface),(ii) a second portion 464 connecting the first portion 462 to theceiling 472 and angled to project energy towards the center of the room(i.e., a first side surface), and (iii) a third portion 464 connectingthe first portion 462 to the floor 474 and angled to project energytowards the center of the room (i.e., a second side surface). The firstportion is a vertical plane in the room and has a horizontal and avertical axis. The second and third portions are angled towards thecenter of the room along the vertical axis.

In this example, the viewing volume 468B of the LF display system 450Bis in the center of the room and partially surrounded by the threeportions of the aggregate surface 460. Similar to the configurationshown in FIG. 4C, the two side portions (e.g., second portion 464, andthird portion 466) are angled to surround the viewer and form asurrounding surface. The surrounding surface increases the viewing FOVfrom the perspective of any viewer in the holographic viewing volume468B. Additionally, the surrounding surface allows the viewing volume468B to be closer to the surface of the displays such that projectedobjects appear closer. In other words, the angled placement of the sidesurfaces increases the field of view, decreases the separation, andincreases the proximity of the aggregate surface, thereby increasing theimmersive experience for viewers. Further, as will be discussed below,deflection optics may be used to optimize the size and position of theviewing volume 468B.

The sloped configuration of the side portions of the aggregate surface460 enables holographic content to be presented closer to the viewingvolume 468B than if the third portion 466 was not sloped. For example,the lower extremities (e.g., legs) of a character presented form a LFdisplay system in a sloped configuration may seem closer and morerealistic than if a LF display system with a flat front wall were used.

Additionally, the configuration of the LF display system and theenvironment which it is located may inform the shape and locations ofthe viewing volumes and viewing sub-volumes.

FIG. 4E, for example, illustrates a top down view of a LF display system450C with an aggregate surface 460 on a front wall 452 of a room. Inthis example, the LF display system 450D is located in a room with afront wall 452, a rear wall 454, a first sidewall 456, a second sidewall458, a ceiling (not shown), and a floor (not shown).

LF display system 450C projects various rays from the aggregate surface460. From each position on the display surface, light rays are projectedin an angular range that is centered on the viewing volume. The raysprojected from the left side of the aggregate surface 460 havehorizontal angular range 481, rays projected from the right side of theaggregate surface have horizontal angular range 482, and rays projectedfrom the center of the aggregate surface 460 have horizontal angularrange 483. In between these points, the projected rays may take onintermediate values of angle ranges as described below in regard to FIG.6 . Having a gradient deflection angle in the projected rays across thedisplay surface in this manner creates a viewing volume 468C. Further,this configuration avoids wasting resolution of the display onprojecting rays into the side walls 456 and 458.

FIG. 4F illustrates a side view of a LF display system 450D with anaggregate surface 460 on a front wall 452 of a room. In this example,the LF display system 450E is located in a room with a front wall 452, arear wall 454, a first sidewall (not shown), a second sidewall (notshown), a ceiling 472, and a floor 474. In this example, the floor istiered such that each tier rises in steps moving from the front wall tothe back wall. Here, each tier of the floor includes a viewingsub-volume (e.g., viewing sub volume 470A and 470B). A tiered floorallows for viewing sub-volumes that do not overlap. That is, eachviewing sub-volume has a line of sight from the viewing sub-volume tothe aggregate surface 460 that does not pass through another viewingsub-volume. In other words, this orientation produces a “stadiumseating” effect in which the vertical offset between tiers allows anunobstructed line of sight which allows each tier to “see over” theviewing sub-volumes of other tiers. LF display systems including viewingsub-volumes that do not overlap may provide a higher quality viewingexperience than LF display systems that have viewing volumes that dooverlap. For example, in the configuration shown in FIG. 4F, differentholographic content may be projected to the audiences in viewingsub-volumes 470A and 470B.

Control of a LF Display System

FIG. 5A is a block diagram of a LF display system 500, in accordancewith one or more embodiments. The LF display system 500 comprises a LFdisplay assembly 510 and a controller 520. The LF display assembly 510includes one or more LF display modules 512 which project a light field.A LF display module 512 may include a source/sensor system 514 thatincludes an integrated energy source(s) and/or energy sensor(s) whichproject and/or sense other types of energy. The controller 520 includesa datastore 522, a network interface 524, and a LF processing engine530. The controller 520 may also include a tracking module 526, and aviewer profiling module 528. In some embodiments, the LF display system500 also includes a sensory feedback system 570 and a tracking system580. The LF display systems described in the context of FIGS. 1, 2, 3,and 4 are embodiments of the LF display system 500. In otherembodiments, the LF display system 500 comprises additional or fewermodules than those described herein. Similarly, the functions can bedistributed among the modules and/or different entities in a differentmanner than is described here. Applications of the LF display system 500are also discussed in detail below with regard to FIGS. 6-9 .

The LF display assembly 510 provides holographic content in aholographic object volume that may be visible to viewers located withina viewing volume. The LF display assembly 510 may provide holographiccontent by executing display instructions received from the controller520. The holographic content may include one or more holographic objectsthat are projected in front of an aggregate surface the LF displayassembly 510, behind the aggregate surface of the LF display assembly510, or some combination thereof. Generating display instructions withthe controller 520 is described in more detail below.

The LF display assembly 510 provides holographic content using one ormore LF display modules (e.g., any of the LF display module 110, the LFdisplay system 200, and LF display module 300) included in an LF displayassembly 510. For convenience, the one or more LF display modules may bedescribed herein as LF display module 512. The LF display module 512 canbe tiled to form a LF display assembly 510. The LF display modules 512may be structured as various seamless surface environments (e.g., singlesided, multi-sided, a wall of a presentation space, a curved surface,etc.). That is, the tiled LF display modules form an aggregate surface.As previously described, a LF display module 512 includes an energydevice layer (e.g., energy device layer 220) and an energy waveguidelayer (e.g., energy waveguide layer 240) that present holographiccontent. The LF display module 512 may also include an energy relaylayer (e.g., energy relay layer 230) that transfers energy between theenergy device layer and the energy waveguide layer when presentingholographic content.

The LF display module 512 may also include other integrated systemsconfigured for energy projection and/or energy sensing as previouslydescribed. For example, a light field display module 512 may include anynumber of energy devices (e.g., energy device 340) configured to projectand/or sense energy. For convenience, the integrated energy projectionsystems and integrated energy sensing systems of the LF display module512 may be described herein, in aggregate, as the source/sensor system514. The source/sensor system 514 is integrated within the LF displaymodule 512, such that the source/sensor system 514 shares the sameseamless energy surface with LF display module 512. In other words, theaggregate surface of an LF display assembly 510 includes thefunctionality of both the LF display module 512 and the source/sensormodule 514. That is, an LF assembly 510 including a LF display module512 with a source/sensor system 514 may project energy and/or senseenergy while simultaneously projecting a light field. For example, theLF display assembly 510 may include a LF display module 512 andsource/sensor system 514 configured as a dual-energy surface orbidirectional energy surface as previously described.

In some embodiments, the LF display system 500 augments the generatedholographic content with other sensory content (e.g., coordinated touch,audio, or smell) using a sensory feedback system 570. The sensoryfeedback system 570 may augment the projection of holographic content byexecuting display instructions received from the controller 520.Generally, the sensory feedback system 570 includes any number ofsensory feedback devices external to the LF display assembly 510 (e.g.,sensory feedback system 442). Some example sensory feedback devices mayinclude coordinated acoustic projecting and receiving devices, aromaprojecting devices, temperature adjustment devices, force actuationdevices, pressure sensors, transducers, etc. In some cases, the sensoryfeedback system 570 may have similar functionality to the light fielddisplay assembly 510 and vice versa. For example, both a sensoryfeedback system 570 and a light field display assembly 510 may beconfigured to generate a sound field. As another example, the sensoryfeedback system 570 may be configured to generate haptic surfaces whilethe light field display 510 assembly is not.

To illustrate, in an example embodiment of a light field display system500, a sensory feedback system 570 may include one or more acousticprojection devices. The one or more acoustic projection devices areconfigured to generate one or more pressure waves that complement theholographic content when executing display instructions received fromthe controller 520. The generated pressure waves may be, e.g., audible(for sound), ultrasonic (for touch), or some combination thereof.Similarly, the sensory feedback system 570 may include an aromaprojecting device. The aroma projecting device may be configured toprovide scents to some, or all, of the target area when executingdisplay instructions received from the controller. The aroma devices maybe tied into an air circulation system (e.g., ducting, fans, or vents)to coordinate air flow within the target area. Further, the sensoryfeedback system 570 may include a temperature adjustment device. Thetemperature adjustment device is configured to increase or decreasetemperature in some, or all, of the target area when executing displayinstructions received from the controller 520.

In some embodiments, the sensory feedback system 570 is configured toreceive input from viewers of the LF display system 500. In this case,the sensory feedback system 570 includes various sensory feedbackdevices for receiving input from viewers. The sensor feedback devicesmay include devices such as acoustic receiving devices (e.g., amicrophone), pressure sensors, joysticks, motion detectors, transducers,etc. The sensory feedback system may transmit the detected input to thecontroller 520 to coordinate generating holographic content and/orsensory feedback.

To illustrate, in an example embodiment of a light field displayassembly, a sensory feedback system 570 includes a microphone. Themicrophone is configured to record audio produced by one or more viewers(e.g., gasps, screams, laughter, etc.). The sensory feedback system 570provides the recorded audio to the controller 520 as viewer input. Thecontroller 520 may use the viewer input to generate holographic content.Similarly, the sensory feedback system 570 may include a pressuresensor. The pressure sensor is configured to measure forces applied byviewers to the pressure sensor. The sensory feedback system 570 mayprovide the measured forces to the controller 520 as viewer input.

In some embodiments, the LF display system 500 includes a trackingsystem 580. The tracking system 580 includes any number of trackingdevices configured to determine the position, movement and/orcharacteristics of viewers in the target area. Generally, the trackingdevices are external to the LF display assembly 510. Some exampletracking devices include a camera assembly (“camera”), a depth sensor,structured light, a LIDAR system, a card scanning system, or any othertracking device that can track viewers within a target area.

The tracking system 580 may include one or more energy sources thatilluminate some or all of the target area with light. However, in somecases, the target area is illuminated with natural light and/or ambientlight from the LF display assembly 510 when presenting holographiccontent. The energy source projects light when executing instructionsreceived from the controller 520. The light may be, e.g., a structuredlight pattern, a pulse of light (e.g., an IR flash), or some combinationthereof. The tracking system may project light in the visible band (˜380nm to 750 nm), in the infrared (IR) band (˜750 nm to 1700 nm), in theultraviolet band (10 nm to 380 nm), some other portion of theelectromagnetic spectrum, or some combination thereof. A source mayinclude, e.g., a light emitted diode (LED), a micro LED, a laser diode,a TOF depth sensor, a tunable laser, etc.

The tracking system 580 may adjust one or more emission parameter whenexecuting instructions received from the controller 520. An emissionparameter is a parameter that affects how light is projected from asource of the tracking system 580. An emission parameter may include,e.g., brightness, pulse rate (to include continuous illumination),wavelength, pulse length, some other parameter that affects how light isprojected from the source assembly, or some combination thereof. In oneembodiment, a source projects pulses of light in a time-of-flightoperation.

The camera of the tracking system 580 captures images of the light(e.g., structured light pattern) reflected from the target area. Thecamera captures images when executing tracking instructions receivedfrom the controller 520. As previously described, the light may beprojected by a source of the tracking system 580. The camera may includeone or more cameras. That is, a camera may be, e.g., an array (1D or 2D)of photodiodes, a CCD sensor, a CMOS sensor, some other device thatdetects some or all of the light project by the tracking system 580, orsome combination thereof. In an embodiment, the tracking system 580 maycontain a light field camera external to the LF display assembly 510. Inother embodiments, the cameras are included as part of the LF displaysource/sensor module 514 included in the LF display assembly 510. Forexample, as previously described, if the energy relay element of a lightfield module 512 is a bidirectional energy layer which interleaves bothemissive displays and imaging sensors at the energy device layer 220,the LF display assembly 510 can be configured to simultaneously projectlight fields and record imaging information from the viewing area infront of the display. In one embodiment, the captured images from thebidirectional energy surface form a light field camera. The cameraprovides captured images to the controller 520.

The camera of the tracking system 580 may adjust one or more imagingparameters when executing tracking instructions received from thecontroller 520. An imaging parameter is a parameter that affects how thecamera captures images. An imaging parameter may include, e.g., framerate, aperture, gain, exposure length, frame timing, rolling shutter orglobal shutter capture modes, some other parameter that affects how thecamera captures images, or some combination thereof.

The controller 520 controls the LF display assembly 510 and any othercomponents of the LF display system 500. The controller 520 comprises adata store 522, a network interface 524, a tracking module 526, a viewerprofiling module 528, and a light field processing engine 530. In otherembodiments, the controller 520 comprises additional or fewer modulesthan those described herein. Similarly, the functions can be distributedamong the modules and/or different entities in a different manner thanis described here. For example, the tracking module 526 may be part ofthe LF display assembly 510 or the tracking system 580.

The data store 522 is a memory that stores information for the LFdisplay system 500. The stored information may include displayinstructions, tracking instructions, emission parameters, imagingparameters, a virtual model of a target area, tracking information,images captured by the camera, one or more viewer profiles, calibrationdata for the light field display assembly 510, configuration data forthe LF display system 510 including resolution and orientation of LFmodules 512, desired viewing volume geometry, content for graphicscreation including 3D models, scenes and environments, materials andtextures, other information that may be used by the LF display system500, or some combination thereof. The data store 522 is a memory, suchas a read only memory (ROM), dynamic random access memory (DRAM), staticrandom access memory (SRAM), or some combination thereof.

The network interface 524 allows the light field display system tocommunicate with other systems or environments via a network. In oneexample, the LF display system 500 receives holographic content from aremote light field display system via the network interface 524. Inanother example, the LF display system 500 transmits holographic contentto a remote data store using the network interface 524.

The tracking module 526 tracks viewers viewing content presented by theLF display system 500. To do so, the tracking module 526 generatestracking instructions that control operation of the source(s) and/or thecamera(s) of the tracking system 580, and provides the trackinginstructions to the tracking system 580. The tracking system 580executes the tracking instructions and provides tracking input to thetracking module 526.

The tracking module 526 may determine a position of one or more viewerswithin the target area (e.g., sitting in the seats of a presentationspace). The determined position may be relative to, e.g., some referencepoint (e.g., a display surface). In other embodiments, the determinedposition may be within the virtual model of the target area. The trackedposition may be, e.g., the tracked position of a viewer and/or a trackedposition of a portion of a viewer (e.g., eye location, hand location,etc.). The tracking module 526 determines the position using one or morecaptured images from the cameras of the tracking system 580. The camerasof the tracking system 580 may be distributed about the LF displaysystem 500, and can capture images in stereo, allowing for the trackingmodule 526 to passively track viewers. In other embodiments, thetracking module 526 actively tracks viewers. That is, the trackingsystem 580 illuminates some portion of the target area, images thetarget area, and the tracking module 526 uses time of flight and/orstructured light depth determination techniques to determine position.The tracking module 526 generates tracking information using thedetermined positions.

The tracking module 526 may also receive tracking information as inputsfrom viewers of the LF display system 500. The tracking information mayinclude body movements that correspond to various input options that theviewer is provided by the LF display system 500. For example, thetracking module 526 may track a viewer's body movement and assign anyvarious movement as an input to the LF processing engine 530. Thetracking module 526 may provide the tracking information to the datastore 522, the LF processing engine 530, the viewer profiling module528, any other component of the LF display system 500, or somecombination thereof.

To provide context for the tracking module 526, consider an exampleembodiment of an LF display system 500 that displays a play in which aperformer in the play scores a winning touchdown. In response to thescene, a viewer first pumps the air to show their excitement. Thetracking system 580 may record the movement of the viewer's hands andtransmit the recording to the tracking module 526. This may be achievedwith a tracking system 580 comprised of cameras, depth sensors, or otherdevices that are external to the light field display assembly 510, orwith a display surface which simultaneously projects light field imagesand records images, wherein the images recorded from the display surfacemay be light field images, or any combination of these devices, aspreviously described. The tracking module 526 tracks the motion of theviewer's hands in the recording and sends the input to LF processingengine 530. The viewer profiling module 528, as described below,determines that information in the image indicates that motion of theviewer's hands are associated with a positive response. Accordingly, ifenough viewers are recognized having a positive response, the LFprocessing engine 530 generates appropriate holographic content tocelebrate the touchdown. For example, the LF processing engine 530 mayproject confetti in the scene.

The LF display system 500 includes a viewer profiling module 528configured to identify and profile viewers. The viewer profiling module528 generates a profile of a viewer (or viewers) that views holographiccontent displayed by a LF display system 500. The viewer profilingmodule 528 generates a viewer profile based, in part, on viewer inputand monitored viewer behavior, actions, and reactions. The viewerprofiling module 528 can access information obtained from trackingsystem 580 (e.g., recorded images, videos, sound, etc.) and process thatinformation to determine various information. In various examples,viewer profiling module 528 can use any number of machine vision ormachine hearing algorithms to determine viewer behavior, actions, andreactions. Monitored viewer behavior can include, for example, smiles,cheering, clapping, laughing, fright, screams, excitement levels,recoiling, other changes in gestures, or movement by the viewers, etc.

More generally, a viewer profile may include any information receivedand/or determined about a viewer viewing holographic content from the LFdisplay system. For example, each viewer profile may log actions orresponses of that viewer to the content displayed by the LF displaysystem 500. Some example information that can be included in a viewerprofile are provided below.

In some embodiments, a viewer profile may describe a response of aviewer within the presentation space with respect to a person displayedin the holographic content (e.g., an athlete, a player, etc.). Forexample, a viewer profile may indicate that a viewer generally haspositive response to athletes of particular teams that have a mascot ofa Cowboy.

In some embodiments, a viewer profile can indicate characteristics of aviewer viewing a sporting event. For example, a viewer in a presentationspace is wearing a sweatshirt displaying a university logo. In thiscase, the viewer profile can indicate that the viewer is wearing asweatshirt and may prefer holographic content associated with theuniversity whose logo is on the sweatshirt. More broadly, viewercharacteristics that can be indicated in a viewer profile may include,for example, age, sex, ethnicity, clothing, viewing location in thepresentation space, etc.

In some embodiments, a viewer profile can indicate preferences for aviewer in regard to desirable sporting event and/or presentation spacecharacteristics. For example, a viewer profile may indicate that aviewer prefers only to view holographic content that is age appropriatefor everyone in their family. In another example, a viewer profile mayindicate holographic object volumes to display holographic content(e.g., on a wall) and holographic object volumes to not displayholographic content (e.g., above their head). The viewer profile mayalso indicate that the viewer prefers to have haptic interfacespresented near them, or prefers to avoid them.

In another example, a viewer profile indicates a history of sportingevents viewed for a particular viewer. For instance, viewer profilingmodule 528 determines that a viewer, or group of viewers, has previouslyattended a sporting event. As such the LF display system 500 may displayholographic content that is different than the previous time the viewersattended the sporting event. As one example, a sporting event includingholographic content may have three different holographic advertisementsduring game intermissions, and LF display system 500 may displaydifferent advertisements based on the viewers in attendance. In anotherexample, each of the three advertisements may be presented to differentviewing volumes in the same presentation space.

In some embodiments, a viewer profile may also describe characteristicsand preferences for a group of viewers rather than a particular viewer.For example, viewer profiling module 528 may generate a viewer profilefor the audience viewing a sporting event in the presentation space. Inone example, viewer profiling module 528 creates a viewer profile forviewers viewing a sporting event such as beach volleyball. The profileindicates that 54.3% of the viewers are women between the age of 20 and35 and have a positive response to the sporting event. The profile alsoindicates that the remaining 46.7% of the viewers are men between theages of 20 and 35 and are having a mediocre response to the sportingevent. Any of the previously described information and characteristicsmay be applied to a group of viewers.

The viewer profiling module 528 may also access a profile associatedwith a particular viewer (or viewers) from a third-party system orsystems to build a viewer profile. For example, a viewer purchases aticket for a sporting event using a third-party vendor that is linked tothat viewer's social media account. Thus, the viewer's ticket is linkedto his social media account. When the viewer enters a presentation spacefor the sporting event using their ticket, the viewer profiling module528 can access information from his social media account to build (oraugment) a viewer profile.

In some embodiments, the data store 522 includes a viewer profile storethat stores viewer profiles generated, updated, and/or maintained by theviewer profiling module 528. The viewer profile can be updated in thedata store at any time by the viewer profiling module 528. For example,in an embodiment, the viewer profile store receives and storesinformation regarding a particular viewer in their viewer profile whenthe particular viewer views holographic content provided by the LFdisplay system 500. In this example, the viewer profiling module 528includes a facial recognition algorithm that may recognize viewers andpositively identify them as they view presented holographic content. Toillustrate, as a viewer enters the target area of the LF display system500, the tracking system 580 obtains an image of the viewer. The viewerprofiling module 528 inputs the captured image and identifies theviewer's face using the facial recognition algorithm. The identifiedface is associated with a viewer profile in the profile store and, assuch, all input information obtained about that viewer may be stored intheir profile. The viewer profiling module may also utilize cardidentification scanners, voice identifiers, a radio-frequencyidentification (RFID) chip scanners, barcode scanners, etc. topositively identify a viewer.

In embodiments where the viewer profiling module 528 can positivelyidentify viewers, the viewer profiling module 528 can determine eachvisit of each viewer to the LF display system 500. The viewer profilingmodule 528 may then store the time and date of each visit in the viewerprofile for each viewer. Similarly, the viewer profiling module 528 maystore received inputs from a viewer from any combination of the sensoryfeedback system 570, the tracking system 580, and/or the LF displayassembly 510 each time they occur. The viewer profile system 528 mayadditionally receive further information about a viewer from othermodules or components of the controller 520 which can then be storedwith the viewer profile. Other components of the controller 520 may thenalso access the stored viewer profiles for determining subsequentcontent to be provided to that viewer.

The LF processing engine 530 generates holographic content comprised oflight field data, as well as data for all of the sensory domainssupported by a LF display system 500. For example, LF processing engine530 may generate 4D coordinates in a rasterized format (“rasterizeddata”) that, when executed by the LF display assembly 510, cause the LFdisplay assembly 510 to present holographic content. The LF processingengine 530 may access the rasterized data from the data store 522.Additionally, the LF processing engine 530 may construct rasterized datafrom a vectorized data set. Vectorized data is described below. The LFprocessing engine 530 can also generate sensory instructions required toprovide sensory content that augments the holographic objects. Asdescribed above, sensory instructions may generate, when executed by theLF display system 500, haptic surfaces, sound fields, and other forms ofsensory energy supported by the LF display system 500. The LF processingengine 530 may access sensory instructions from the data store 522, orconstruct the sensory instructions form a vectorized data set. Inaggregate, the 4D coordinates and sensory data represent holographiccontent as display instructions executable by a LF display system togenerate holographic and sensory content. More generally, holographiccontent can take the form of CG content with ideal light fieldcoordinates, live action content, rasterized data, vectorized data,electromagnetic energy transported by a set of relays, instructions sentto a group of energy devices, energy locations on one or more energysurfaces, the set of energy propagation paths that are projected fromthe display surface, a holographic object that is visible to a viewer oran audience, and many other similar forms.

The amount of rasterized data describing the flow of energy through thevarious energy sources in a LF display system 500 is incredibly large.While it is possible to display the rasterized data on a LF displaysystem 500 when accessed from a data store 522, it is untenable toefficiently transmit, receive (e.g., via a network interface 524), andsubsequently display the rasterized data on a LF display system 500.Take, for example, rasterized data representing a short sporting eventfor holographic projection by a LF display system 500. In this example,the LF display system 500 includes a display containing severalgigapixels and the rasterized data contains information for each pixellocation on the display. The corresponding size of the rasterized datais vast (e.g., many gigabytes per second of sporting event displaytime), and unmanageable for efficient transfer over commercial networksvia a network interface 524. The efficient transfer problem may beamplified for applications including live streaming of holographiccontent. An additional problem with merely storing rasterized data ondata store 522 arises when an interactive experience is desired usinginputs from the sensory feedback system 570 or the tracking module 526.To enable an interactive experience, the light field content generatedby the LF processing engine 530 can be modified in real-time in responseto sensory or tracking inputs. In other words, in some cases, LF contentcannot simply be read from the data store 522.

Therefore, in some configurations, data representing holographic contentfor display by a LF display system 500 may be transferred to the LFprocessing engine 530 in a vectorized data format (“vectorized data”).Vectorized data may be orders of magnitude smaller than rasterized data.Further, vectorized data provides high image quality while having a dataset size that enables efficient sharing of the data. For example,vectorized data may be a sparse data set derived from a denser data set.Thus, vectorized data may have an adjustable balance between imagequality and data transmission size based on how sparse vectorized datais sampled from dense rasterized data. Tunable sampling to generatevectorized data enables optimization of image quality for a givennetwork speed. Consequently, vectorized data enables efficienttransmission of holographic content via a network interface 524.Vectorized data also enables holographic content to be live-streamedover a commercial network.

In summary, the LF processing engine 530 may generate holographiccontent derived from rasterized data accessed from the data store 522,vectorized data accessed from the data store 522, or vectorized datareceived via the network interface 524. In various configurations,vectorized data may be encoded by an encoder before data transmission,and decoded by a decoder within the LF controller 520 after reception.The encoder and decoder pair may be part of the same proprietary systemcodec. In some examples, the vectorized data is encoded for added datasecurity and sporting event improvements related to data compression.For example, vectorized data received by the network interface may beencoded vectorized data received from a holographic streamingapplication. In some examples, vectorized data may require a decoder,the LF processing engine 530, or both of these to access informationcontent encoded in vectorized data. The encoder and/or decoder systemsmay be available to customers or licensed to third-party vendors. Otherexample encoding and/or decoding schemes can be employed to transmitand/or present holographic content.

Vectorized data contains all the information for each of the sensorydomains supported by a LF display system 500 in way that may support aninteractive experience. For example, vectorized data for an interactiveholographic experience may include any vectorized properties that canprovide accurate physics for each of the sensory domains supported by aLF display system 500. Vectorized properties may include any propertiesthat can be synthetically programmed, captured, computationallyassessed, etc. A LF processing engine 530 may be configured to translatevectorized properties in vectorized data to rasterized data. The LFprocessing engine 530 may then project holographic content translatedfrom the vectorized data using the LF display assembly 510. In variousconfigurations, the vectorized properties may include one or morered/green/blue/alpha channel (RGBA)+depth images, multi view images withor without depth information at varying resolutions that may include onehigh-resolution center image and other views at a lower resolution,material properties such as albedo and reflectance, surface normals,other optical effects, surface identification, geometrical objectcoordinates, virtual camera coordinates, display plane locations,lighting coordinates, tactile stiffness for surfaces, tactile ductility,tactile strength, amplitude and coordinates of sound fields,environmental conditions, somatosensory energy vectors related to themechanoreceptors for textures or temperature, audio, and any othersensory domain property. Many other vectorized properties are alsopossible.

The LF display system 500 may also generate an interactive viewingexperience. That is, holographic content may be responsive to inputstimuli containing information about viewer locations, gestures,interactions, interactions with holographic content, or otherinformation derived from the viewer profiling module 528, and/ortracking module 526. For example, in an embodiment, a LF processingsystem 500 creates an interactive viewing experience using vectorizeddata of a real-time sporting event received via a network interface 524.In another example, if a holographic object needs to move in a certaindirection immediately in response to a viewer interaction, the LFprocessing engine 530 may update the render of the scene so theholographic object moves in that required direction. This may requirethe LF processing engine 530 to use a vectorized data set to renderlight fields in real time based a 3D graphical scene with the properobject placement and movement, collision detection, occlusion, color,shading, lighting, etc., correctly responding to the viewer interaction.The LF processing engine 530 converts the vectorized data intorasterized data for presentation by the LF display assembly 510. The LFdisplay system 500 may employ various other encoding/decoding techniquesthat allow the LF display system to present holographic content in anapproximately real time.

The rasterized data includes holographic content instructions andsensory instructions (display instructions) representing the real-timesporting event. The LF display assembly 510 simultaneously projectsholographic and sensory content of the real-time sporting event byexecuting the display instructions. The LF display system 500 monitorsviewer interactions (e.g., vocal response, touching, etc.) with thepresented real-time sporting event with the tracking module 526 andviewer profiling module 528. In response to the viewer interactions, theLF processing engine may create an interactive experience by generatingadditional holographic and/or sensory content for display to theviewers.

To illustrate, consider an example embodiment of an LF display system500 including a LF processing engine 530 that generates a plurality ofholographic objects representing balloons falling from the ceiling of apresentation space during a sporting event. A viewer may move to touchthe holographic object representing the balloon. Correspondingly, thetracking system 580 tracks movement of the viewer's hands relative tothe holographic object. The movement of the viewer is recorded by thetracking system 580 and sent to the controller 520. The tracking module526 continuously determines the motion of the viewer's hand and sendsthe determined motions to the LF processing engine 530. The LFprocessing engine 530 determines the placement of the viewer's hand inthe scene, adjusts the real-time rendering of the graphics to includeany required change in the holographic object (such as position, color,or occlusion). The LF processing engine 530 instructs the LF displayassembly 510 (and/or sensory feedback system 570) to generate a tactilesurface using the volumetric haptic projection system (e.g., usingultrasonic speakers). The generated tactile surface corresponds to atleast a portion of the holographic object and occupies substantially thesame space as some or all of an exterior surface of the holographicobject. The LF processing engine 530 uses the tracking information todynamically instruct the LF display assembly 510 to move the location ofthe tactile surface along with a location of the rendered holographicobject such that the viewer is given both a visual and tactileperception of touching the balloon. More simply, when a viewer views hishand touching a holographic balloon, the viewer simultaneously feelshaptic feedback indicating their hand touches the holographic balloon,and the balloon changes position or motion in response to the touch. Insome examples, rather than presenting and interactive balloon in asporting event accessed from a data store 522, the interactive balloonmay be received as part of holographic content received from alive-streaming application via a network interface 524. In other words,the holographic content displayed by the LF display system 500 may be aholographic content livestream.

LF processing engine 530 may provide holographic content to display toviewers in a presentation space before, during, and/or after a sportingevent to augment the presentation space experience. The holographiccontent may be provided by the publisher of the sporting event, providedby the presentation space, provided by an advertiser, generated by a LFprocessing engine 530, etc. The holographic content may be contentassociated with the sporting event, the genre of the sporting event, thelocation of the presentation space, advertisements, etc. In any case,the holographic content may be stored in the data store 522, or streamedto the LF display system 500 in vectorized format through the networkinterface 524. For example, a sporting event may be shown in apresentation space augmented with LF display modules on the walls. Thedistributor of the sporting event may provide holographic content topresent on the wall displays before the sporting event begins. The LFprocessing engine 530 accesses the holographic content and presents theaccessed content from the displays on the walls of the presentationspace before the sporting event begins. In another example, apresentation space with an LF display system 500 is located in SanFrancisco. The LF display system of a presentation space stores aholographic representation of the Golden Gate Bridge to present before asporting event if no sporting event specific content is provided. Here,as no sporting event-specific holographic content is provided, the LFprocessing engine 530 accesses and presents the Golden Gate Bridge inthe presentation space. In another example, an advertiser has providedholographic content of its products as advertisements to a presentationspace to display after a sporting event. After the sporting eventconcludes, the LF processing engine 530 presents the advertisements tothe viewers as they leave the presentation space. In other examples, asdescribed below, a LF processing engine may dynamically generateholographic content to display on the walls of the theater.

The LF processing engine 500 may also modify holographic content to suitthe presentation space that is presenting the holographic content. Forexample, not every presentation space is the same size, has the samenumber of seats, or has the same technical configuration. As such, LFprocessing engine 530 may modify holographic content such that it willbe appropriately displayed in a presentation space. In an embodiment,the LF processing engine 530 may access a configuration file of apresentation space including the layout, resolution, field-of-view,other technical specifications, etc. of the presentation space. The LFprocessing engine 530 may render and present the holographic contentbased on information included in the configuration file.

The LF processing engine 530 may also create holographic content fordisplay by the LF display system 500. Importantly, here, creatingholographic content for display is different from accessing, orreceiving, holographic content for display. That is, when creatingcontent, the LF processing engine 530 generates entirely new content fordisplay rather than accessing previously generated and/or receivedcontent. The LF processing engine 530 can use information from thetracking system 580, the sensory feedback system 570, the viewerprofiling module 528, the tracking module 526, or some combinationthereof, to create holographic content for display. In some examples, LFprocessing engine 530 may access information from elements of the LFdisplay system 500 (e.g., tracking information and/or a viewer profile),create holographic content based on that information, and display thecreated holographic content using the LF display system 500 in response.The created holographic content may be augmented with other sensorycontent (e.g., touch, audio, or smell) when displayed by the LF displaysystem 500. Further, the LF display system 500 may store createdholographic content such that it may be displayed in the future.

Dynamic Content Generation for a LF Display System

In some embodiments, the LF processing engine 530 incorporates anartificial intelligence (AI) model to create holographic content fordisplay by the LF display system 500. The AI model may includesupervised or unsupervised learning algorithms including but not limitedto regression models, neural networks, classifiers, or any other AIalgorithm. The AI model may be used to determine viewer preferencesbased on viewer information recorded by the LF display system 500 (e.g.,by tracking system 580) which may include information on a viewer'sbehavior.

The AI model may access information from the data store 522 to createholographic content. For example, the AI model may access viewerinformation from a viewer profile or profiles in the data store 522 ormay receive viewer information from the various components of the LFdisplay system 500. To illustrate, the AI model may determine a viewerenjoys seeing holographic content in which a performer wears a bow tie.The AI model may determine the preference based on a group of viewer'spositive reactions or responses to previously viewed holographic contentincluding a bow-tie wearing actor. That is, the AI model may createholographic content personalized to a set of viewers according to thelearned preferences of those viewers. So, for example, the AI model maycreate bow-ties for actors displayed in the holographic content viewedby a group of viewers using the LF display system 500. The AI model mayalso store the learned preferences of each viewer in the viewer profilestore of the data store 522. In some examples, the AI model may createholographic content for an individual viewer rather than a group ofviewers.

One example of an AI model that can be used to identify characteristicsof viewers, identify reactions, and/or generate holographic contentbased on the identified information is a convolutional neural networkmodel with layers of nodes, in which values at nodes of a current layerare a transformation of values at nodes of a previous layer. Atransformation in the model is determined through a set of weights andparameters connecting the current layer and the previous layer. Forexample, and AI model may include five layers of nodes: layers A, B, C,D, and E. The transformation from layer A to layer B is given by afunction W₁, the transformation from layer B to layer C is given by W₂,the transformation from layer C to layer D is given by W₃, and thetransformation from layer D to layer E is given by W₄. In some examples,the transformation can also be determined through a set of weights andparameters used to transform between previous layers in the model. Forexample, the transformation W₄ from layer D to layer E can be based onparameters used to accomplish the transformation W₁ from layer A to B.

The input to the model can be an image taken by tracking system 580encoded onto the convolutional layer A and the output of the model isholographic content decoded from the output layer E. Alternatively oradditionally, the output may be a determined characteristic of a viewerin the image. In this example, the AI model identifies latentinformation in the image representing viewer characteristics in theidentification layer C. The AI model reduces the dimensionality of theconvolutional layer A to that of the identification layer C to identifyany characteristics, actions, responses, etc. in the image. In someexamples, the AI model then increases the dimensionality of theidentification layer C to generate holographic content.

The image from the tracking system 580 is encoded to a convolutionallayer A. Images input in the convolutional layer A can be related tovarious characteristics and/or reaction information, etc. in theidentification layer C. Relevance information between these elements canbe retrieved by applying a set of transformations between thecorresponding layers. That is, a convolutional layer A of an AI modelrepresents an encoded image, and identification layer C of the modelrepresents a smiling viewer. Smiling viewers in a given image may beidentified by applying the transformations W₁ and W₂ to the pixel valuesof the image in the space of convolutional layer A. The weights andparameters for the transformations may indicate relationships betweeninformation contained in the image and the identification of a smilingviewer. For example, the weights and parameters can be a quantization ofshapes, colors, sizes, etc. included in information representing asmiling viewer in an image. The weights and parameters may be based onhistorical data (e.g., previously tracked viewers).

Smiling viewers in the image are identified in the identification layerC. The identification layer C represents identified smiling viewersbased on the latent information about smiling viewers in the image.

Identified smiling viewers in an image can be used to generateholographic content. To generate holographic content, the AI modelstarts at the identification layer C and applies the transformations W₂and W₃ to the value of the given identified smiling viewers in theidentification layer C. The transformations result in a set of nodes inthe output layer E. The weights and parameters for the transformationsmay indicate relationships between an identified smiling viewers andspecific holographic content and/or preferences. In some cases, theholographic content is directly output from the nodes of the outputlayer E, while in other cases the content generation system decodes thenodes of the output layer E into a holographic content. For example, ifthe output is a set of identified characteristics, the LF processingengine can use the characteristics to generate holographic content.

Additionally, the AI model can include layers known as intermediatelayers. Intermediate layers are those that do not correspond to animage, identifying characteristics/reactions, etc., or generatingholographic content. For example, in the given example, layer B is anintermediate layer between the convolutional layer A and theidentification layer C. Layer D is an intermediate layer between theidentification layer C and the output layer E. Hidden layers are latentrepresentations of different aspects of identification that are notobserved in the data, but may govern the relationships between theelements of an image when identifying characteristics and generatingholographic content. For example, a node in the hidden layer may havestrong connections (e.g., large weight values) to input values andidentification values that share the commonality of “laughing peoplesmile.” As another example, another node in the hidden layer may havestrong connections to input values and identification values that sharethe commonality of “scared people scream.” Of course, any number oflinkages are present in a neural network. Additionally, eachintermediate layer is a combination of functions such as, for example,residual blocks, convolutional layers, pooling operations, skipconnections, concatenations, etc. Any number of intermediate layers Bcan function to reduce the convolutional layer to the identificationlayer and any number of intermediate layers D can function to increasethe identification layer to the output layer.

In one embodiment, the AI model includes deterministic methods that havebeen trained with reinforcement learning (thereby creating areinforcement learning model). The model is trained to increase thequality of the sporting event using measurements from tracking system580 as inputs, and changes to the created holographic content asoutputs.

Reinforcement learning is a machine learning system in which a machinelearns ‘what to do’—how to map situations to actions—so as to maximize anumerical reward signal. The learner (e.g. LF processing engine 530) isnot told which actions to take (e.g., generating prescribed holographiccontent), but instead discovers which actions yield the most reward(e.g., increasing the quality of holographic content by making morepeople cheer) by trying them. In some cases, actions may affect not onlythe immediate reward but also the next situation and, through that, allsubsequent rewards. These two characteristics—trial-and-error search anddelayed reward—are two distinguishing features of reinforcementlearning.

Reinforcement learning is defined not by characterizing learningmethods, but by characterizing a learning problem. Basically, areinforcement learning system captures those important aspects of theproblem facing a learning agent interacting with its environment toachieve a goal. That is, in the example of generating a song for aperformer, the reinforcement learning system captures information aboutviewers in the presentation space (e.g., age, disposition, etc.). Suchan agent senses the state of the environment and takes actions thataffect the state to achieve a goal or goals (e.g., creating a pop songfor which the viewers will cheer). In its most basic form, theformulation of reinforcement learning includes three aspects for thelearner: sensation, action, and goal. Continuing with the song example,the LF processing engine 530 senses the state of the environment withsensors of the tracking system 580, displays holographic content to theviewers in the environment, and achieves a goal that is a measure of theviewer's reception of that song.

One of the challenges that arises in reinforcement learning is thetrade-off between exploration and exploitation. To increase the rewardin the system, a reinforcement learning agent prefers actions that ithas tried in the past and found to be effective in producing reward.However, to discover actions that produce reward, the learning agentselects actions that it has not selected before. The agent ‘exploits’information that it already knows in order to obtain a reward, but italso ‘explores’ information in order to make better action selections inthe future. The learning agent tries a variety of actions andprogressively favors those that appear to be best while still attemptingnew actions. On a stochastic task, each action is generally tried manytimes to gain a reliable estimate to its expected reward. For example,if the LF processing engine creates holographic content that the LFprocessing engine knows will result in a viewer laughing after a longperiod of time, the LF processing engine may change the holographiccontent such that the time until a viewer laughs decreases.

Further, reinforcement learning considers the whole problem of agoal-directed agent interacting with an uncertain environment.Reinforcement learning agents have explicit goals, can sense aspects oftheir environments, and can choose actions to receive high rewards(i.e., a roaring crowd). Moreover, agents generally operate despitesignificant uncertainty about the environment they face. Whenreinforcement learning involves planning, the system addresses theinterplay between planning and real-time action selection, as well asthe question of how environmental elements are acquired and improved.For reinforcement learning to make progress, important sub problems haveto be isolated and studied, the sub problems playing clear roles incomplete, interactive, goal-seeking agents.

The reinforcement learning problem is a framing of a machine learningproblem where interactions are processed and actions are carried out toachieve a goal. The learner and decision-maker is called the agent(e.g., LF processing engine 530). The thing it interacts with,comprising everything outside the agent, is called the environment(e.g., viewers in a presentation space, etc.). These two interactcontinually, the agent selecting actions (e.g., creating holographiccontent) and the environment responding to those actions and presentingnew situations to the agent. The environment also gives rise to rewards,special numerical values that the agent tries to maximize over time. Inone context, the rewards act to maximize viewer positive reactions toholographic content. A complete specification of an environment definesa task which is one instance of the reinforcement learning problem.

To provide more context, an agent (e.g., LF processing engine 530) andenvironment interact at each of a sequence of discrete time steps, i.e.t=0, 1, 2, 3, etc. At each time step t the agent receives somerepresentation of the environment's state s_(t) (e.g., measurements fromtracking system 580). The states s t are within S, where S is the set ofpossible states. Based on the state s t and the time step t, the agentselects an action at (e.g., making the performer do the splits). Theaction at is within A(s_(t)), where A(s_(t)) is the set of possibleactions. One time state later, in part as a consequence of its action,the agent receives a numerical reward r_(t+1). The states r_(t+1) arewithin R, where R is the set of possible rewards. Once the agentreceives the reward, the agent selects in a new state s_(t+1).

At each time step, the agent implements a mapping from states toprobabilities of selecting each possible action. This mapping is calledthe agent's policy and is denoted π_(t) where π_(t)(s,a) is theprobability that a_(t)=a if s_(t)=s. Reinforcement learning methods candictate how the agent changes its policy as a result of the states andrewards resulting from agent actions. The agent's goal is to maximizethe total amount of reward it receives over time.

This reinforcement learning framework is flexible and can be applied tomany different problems in many different ways (e.g. generatingholographic content). The framework proposes that whatever the detailsof the sensory, memory, and control apparatus, any problem (orobjective) of learning goal-directed behavior can be reduced to threesignals passing back and forth between an agent and its environment: onesignal to represent the choices made by the agent (the actions), onesignal to represent the basis on which the choices are made (thestates), and one signal to define the agent's goal (the rewards).

Of course, the AI model can include any number of machine learningalgorithms. Some other AI models that can be employed are linear and/orlogistic regression, classification and regression trees, k-meansclustering, vector quantization, etc. Whatever the case, generally, theLF processing engine 530 takes an input from the tracking module 526and/or viewer profiling module 528 and a machine learning model createsholographic content in response. Similarly, the AI model may direct therendering of holographic content.

In an example, the LF processing engine 530 creates a virtual athlete.The LF processing engine 530 creates the virtual athlete usinginformation included in the viewer profiles stored in the data store522. For example, information included in stored viewer profilesindicates that a large number of viewers enjoy high quality soccer froma woman athlete in her early thirties with artificially colored hair. Assuch, the LF processing engine 530 creates an athlete that is displayedby the LF display system 500 as a female soccer player. More explicitly,LF processing engine 530 accesses the viewer profiles of the viewers ina presentation space. LF processing engine 530 parameterizes (e.g.,quantifies) information in each viewer profile. For example, LFprocessing engine 530 can quantify characteristics such as the age,location, sex, etc. of a viewer. Further, LF processing engine 530 canparameterize other information included in a viewer profile. Forexample, if a viewer profile indicates that a viewer has attended foursporting events of female athletes, the content creation module mayquantify this tendency (e.g., generates a score indicating a viewer'sinterest in female athletes). LF processing engine 530 inputs theparameterized user profiles into an AI model (e.g., a neural network)configured to generate characteristics of a virtual athlete based oninput parameters and receives characteristics for the athlete inresponse. LF processing engine 530 then inputs the characteristics forthe virtual athlete into an AI model (e.g., a procedural generationalgorithm) configured to generate an athlete given a set ofcharacteristics and generates a virtual female athlete. Further the LFprocessing engine 530 can create holographic content (e.g., abilities,sporting events, teams, uniforms, etc.) that would adhere to the personaof the virtual athlete. For example, the content generation module 530may create a training regimen for the virtual athlete adherent to herplaying style. More explicitly, LF processing engine 530 may access thecharacteristics of the virtual athlete and information about the viewersand input that information into an AI model (e.g., a recurrent neuralnetwork “RNN”). Again, the characteristics and information may beparameterized (e.g., using a classification and regression tree) andinput into the RNN. Here, the RNN may be trained using training regimenswith similar input parameters. As such, the RNN generates a trainingregimen for the virtual athlete for the viewers in the presentationspace that shares similar characteristics to the training regimens ofother female soccer athletes.

LF processing engine 530 can create holographic content based on asporting event being shown in the presentation space. For example, asporting event being shown in the presentation space may be associatedwith a set of metadata describing the sporting event's characteristics.The metadata may include, for example, the setting, genre, coaches,players, statistics, sporting event type, themes, titles, run-times,etc. LF processing engine 530 may access any of the metadata describingthe sporting event and generate holographic content to present in thepresentation space in response. For example, a sporting event titled“The Excellent Bowl” is a play about to be played in a presentationspace augmented with a LF display system 500. The LF processing engine530 accesses the metadata of the sporting event to create holographiccontent for the walls of the presentation space before the sportingevent begins. Here, the metadata includes a vibrant stadium settingappropriate for The Excellent Bowl. The LF processing engine 530 inputsthe metadata into an AI model and receives holographic content todisplay on the walls of the presentation space in response. In thisexample, the LF processing engine 530 creates a beachside sunset todisplay on the walls of the presentation space before the sporting eventbegins to play.

In an example, the LF processing engine 530 creates holographic contentbased on the viewers present at a presentation space including a LFdisplay system 500. For example, a group of viewers enters apresentation space to view a sporting event augmented by holographiccontent displayed by the LF display system 500. Viewer profiling module528 generates a viewer profile for the viewers in the presentation spaceand an aggregate viewer profile representing all of the viewers in thepresentation space. LF processing engine 530 accesses the aggregateviewer profile and creates holographic content to display to the viewersin the presentation space. For example, the viewers in a presentationspace are a group of fans viewing a team working towards a championship,and, therefore, the aggregate viewer profile includes informationindicating that they may enjoy holographic content commensurate withsuper-fans for a sports team (e.g., through parameterization and inputinto an AI model). As such, the LF processing engine 530 generatesholographic content such that the presentation space is a more rowdyatmosphere (e.g., foam fingers, chants, noise makers, etc.).

In some examples, the LF processing engine 530 may create holographiccontent based on previously existing content. Here, previously existingcan be a previously existing team, athlete, etc. For example, a viewer'sfavorite athlete is “Baby Ruthie” and they wish to view a baseball gamestarring their idol. As such, LF processing engine 530 creates augmentedcontent for a sporting event of the Babie Ruthie smashing home run afterhome run. In this example, LF processing engine 530 may access existingbox-scores, images, and statistics of Babie Ruthie (e.g., stored in datastore 522) and use that content to create sporting event content fordisplay by the LF display system 500. More explicitly, the contentcreation system can access the previous content from the data store 522.The LF processing engine 530 inputs favorite athletes, favorite teams,and favorite plays, and any other viewer data into an AI model (e.g., aprocedural generation algorithm) configured to create, for example, acollage of multiple favorite athletes making plays, and in response theAI model outputs this holographic content to show during a sportingevent. In some instances, because LF processing engine 530 is creatingholographic content based on copyrighted content, the viewers and/or theLF processing engine 530 may pay a fee to the copyright holders.

In an example, the LF processing engine 530 creates holographic contentbased on the responses of viewers viewing a sporting event. For example,viewers in a presentation space are viewing a sporting event in apresentation space augmented by a LF display system 500. The trackingmodule 526 and the viewer profiling module 528 monitor the reaction ofthe viewers viewing the sporting event. For example, tracking module 526may obtain images of viewers as they view the sporting event. Trackingmodule 526 identifies the viewer, and viewer profiling module 528 mayuse machine vision algorithms to determine a reaction of the viewerbased on information included in the image. For example, an AI model canbe used to identify if a viewer viewing the sporting event is smilingand, accordingly, viewer profiling module 528 can indicate in the viewerprofile if the viewer has a positive or negative response to thesporting event based on the smile. Other reactions may also bedetermined. The tracking module may determine information about viewersincluding the position of the viewer, a movement of the viewer, agesture of the viewer, an expression of the viewer, an age of theviewer, a sex of the viewer, an ethnicity of the viewer, or a clothingworn by the viewer. This information may be shared with the viewerprofiling module 528 to generate a viewer profile.

The LF processing engine 530 may create holographic content based onpreviously existing or provided advertisement content. That is, forexample, the LF processing engine 530 can request an advertisement froma network system via network interface 524, the network system providesthe holographic content in response, and the LF processing engine 530creates holographic content for display including the advertisement.Some examples of advertisement can include, products, text, videos, etc.Advertisements may be presented to specific viewing volumes based on theviewers in that viewing volume. Similarly, holographic content mayaugment a sporting event with an advertisement (e.g., a productplacement). Most generally, the LF processing engine 530 can createadvertisement content based on any of the characteristics and/orreactions of the viewers in the presentation space as previouslydescribed.

The preceding examples of creating content are not limiting. Mostbroadly, LF processing engine 530 creates holographic content fordisplay to viewers of a LF display system 500. The holographic contentcan be created based on any of the information included in the LFdisplay system 500.

Holographic Content Distribution Networks

FIG. 5B illustrates an example LF sporting event network 550, inaccordance with one or more embodiments. One or more LF display systemsmay be included in the LF sporting event network 550. The LF sportingevent network 550 includes any number of LF display systems (e.g., 500A,500B, and 500C), a LF generation system 554, and a networking system 556that are coupled to each other via a network 552. The network may be asatellite network, an IP-based network, or a cable system network. Inother embodiments, the LF sporting event network 550 comprisesadditional or fewer entities than those described herein. Similarly, thefunctions can be distributed among the different entities in a differentmanner than is described here.

In the illustrated embodiment, the LF sporting event network 550includes LF display systems 500A, 500B, and 500C that may receiveholographic content via the network 552 and display the holographiccontent to viewers. The LF display systems 500A, 500B, and 500C arecollectively referred to as LF display systems 500.

The LF generation system 554 is a system that generates holographiccontent for display in a sporting event presentation space including aLF display system. In other words, the LF generation system isconfigured to generate any of the LF content described herein. Theholographic content may be a sporting event or may be holographiccontent that augments a traditional sporting event.

In one example, the augmenting content is audio content that can bedisplayed concurrently to the holographic content. For example, theaudio content may include an announcer, a music track, a jingle, a soundeffect, or a translation.

In another example, the augmenting content is any of an overlay for asporting area in the sporting arena, an informational overlay, or anaugmentation of holographic content included in a holographic contentlivestream.

In another example, the augmenting content is an advertisementcomprising any holographic, video, audio, or tactile content. In thiscase the advertisement may be based on any of a location of the LFdisplay system presenting the advertisement, the sporting arena wherethe content was recorded and/or is being presented, a sponsor of thesporting event, or a configuration of the LF display system. In somecases, for viewers paying a higher fee for viewing the holographiccontent, viewer advertisements may be inserted into the holographiclivestream.

A LF generation system 554 may include a light field recording assemblycomprised of any number of sensors and/or processors to record energydata of an event, as well as a processing engine configured to convertthis recorded energy data into holographic content. For example, thesensors of the light field recording assembly can include cameras forrecording images, microphones for recording audio, pressure sensors forrecording interactions with objects, etc. In some examples, the lightfield recording assembly of the LF generation system 554 includes one ormore recording modules (e.g., a LF display module configured to recordenergy data from an event, or a simple 2D camera to capture images of anevent) positioned around an area (e.g., a sporting event presentationspace) to record an event from multiple viewpoints. In this case, theprocessing engine of the LF generation system 554 is configured toconvert the energy from multiple viewpoints into holographic content. Insome examples, the light field recording assembly includes two or moretwo-dimensional recording systems, which are used by the processingengine to convert multiple viewpoints of an event into three-dimensionalholographic content. The light field recording assembly can also includeother sensors, such as, for example, depth sensors and/or plenopticcameras.

More broadly, the LF generation system 554 generates holographic contentfor display in a presentation space by using any recorded sensory dataor synthetic data of an event that may be projected by a LF displaysystem when showing a sporting event. For example, the sensory data mayinclude recorded audio, recorded images, recorded interactions withobjects, etc. Many other types of sensory data may be used. Toillustrate, the recorded visual content may include: 3D graphics scenes,3D models, object placement, textures, color, shading, and lighting; 2Dsporting event data which can be converted to a holographic form usingan AI model and a large data set of similar sporting event conversions;multi-view camera data from a camera rig with many cameras with orwithout a depth channel; plenoptic camera data; CG content; or othertypes of recorded sensory data of an event as described herein.

In various examples, the sporting event that is recorded with one ormore sensors over one or more energy domains may be any type of sportingevent. The sporting event presentation space may include one or more ofa field, a portion of an arena, etc.

In some configurations, the LF generation system 554 may use aproprietary encoder to perform the encoding operation that reduces thesensory data recorded for a sporting event into a vectorized data formatas described above. That is, encoding data to vectorized data mayinclude image processing, audio processing, or any other computationsthat may result in a reduced data set that is easier to transmit overthe network 552. The encoder may support formats used by sportingevent-making industry professionals. In other configurations, the LFgeneration system may transmit the sporting event content to the networksystem 556 and or LF display system without encoding the content.

Each LF display system (e.g., 500A, 500B, 500C) may receive the encodeddata from the network 552 via a network interface 524. In this example,each LF display system includes a decoder to decode the encoded LFdisplay data. More explicitly, a LF processing engine 530 generatesrasterized data for the LF display assembly 510 by applying decodingalgorithms provided by the decoder to the received encoded data. In someexamples, the LF processing engine may additionally generate rasterizeddata for the LF display assembly using input from the tracking module526, the viewer profiling module 528, and the sensory feedback system570 as described herein. Rasterized data generated for the LF displayassembly 510 reproduces the holographic content recorded by the LFgeneration system 554. Importantly, each LF display system 500A, 500B,and 500C generates rasterized data suitable for the particularconfiguration of the LF display assembly in terms of geometry,resolution, etc. In some configurations, the encoding and decodingprocess is part of a proprietary encoding/decoding system pair (or‘codec’) which may be offered to display customers or licensed by thirdparties. In some instances, the encoding/decoding system pair may beimplemented as a proprietary API that may offer content creators acommon programming interface.

In some configurations, various systems in the LF sporting event network550 (e.g., LF display system 500, the LF generation system 554, etc.)may have different hardware configurations. Hardware configurations caninclude arrangement of physical systems, energy sources, energy sensors,haptic interfaces, sensory capabilities, resolutions, fields-of-view, LFdisplay module configurations, or any other hardware description of asystem in the LF sporting event network 550. Each hardware configurationmay generate, or utilize, sensory data in different data formats. Assuch, a decoder system may be configured to decode encoded data for theLF display system on which it will be presented. For example, a LFdisplay system (e.g., LF display system 500A) having a first hardwareconfiguration receives encoded data from a LF generation system (e.g.,LF generation system 554) having a second hardware configuration. Thedecoding system accesses information describing the first hardwareconfiguration of the LF display system 500A. The decoding system decodesthe encoded data using the accessed hardware configuration such that thedecoded data can be processed by the LF processing engine 530 of thereceiving LF display system 500A. The LF processing engine 530 generatesand presents rasterized content for the first hardware configurationdespite being recorded in the second hardware configuration. In asimilar manner, holographic content recorded by the LF generation system554 can be presented by any LF display system (e.g., LF display system500B, LF display system 500C) whatever the hardware configurations.Various other aspects that may be included in the hardware configurationmay include: a resolution, a number of projected rays per degree, afield of view, a deflection angle on the display surface, and adimensionality of the display surface, etc. Additionally, the hardwareconfiguration may also include, a number of display panels of the LFdisplay assembly, a relative orientation of the display panels, a heightof the display panels, a width of the display panels, and a layout ofthe display panels.

Similarly, various systems in the LF sporting event network 550 may havedifferent geometric orientations. Geometric orientations reflect thephysical size, layout, and arrangement of the various modules and systemincluded in an LF display system. As such, a decoder system may beconfigured to decode encoded data for the LF display system in thegeometric configuration on which it will be presented. For example, a LFdisplay system (e.g., LF display system 500A) having a first geometricconfiguration receives encoded data from a LF generation system (e.g.,LF generation system 554) having a second geometric configuration. Thedecoding system accesses information describing the first geometricconfiguration of the LF display system 500A. The decoding system decodesthe encoded data using the accessed geometric configuration such thatthe decoded data can be processed by the LF processing engine 530 of thereceiving LF display system 500A. The LF processing engine 530 generatesand presents content for the first geometric configuration despite beingrecorded in the second geometric configuration. In a similar manner,holographic content recorded by the LF generation system 554 can bepresented by any LF display system (e.g., LF display system 500B, LFdisplay system 500C) whatever the geometric configurations. Variousother aspects that may be included in the geometric configuration mayinclude: a number of display panels (or surfaces) of the LF displayassembly, a relative orientation of the display panels.

Similarly, various presentation spaces in the LF sporting event network550 may have different configurations. Presentation space configurationsreflect any of the number and/or position of holographic object volumes,the number and/or position of viewing volumes, and a number and/orposition of viewing locations relative to a LF display system. As such,a decoder system may be configured to decode encoded data for the LFdisplay system in the presentation space in which it will be presented.For example, a LF display system (e.g., LF display system 500A) havingin a first presentation space receives encoded data from a LF generationsystem (e.g., LF generation system 554) recorded in a differentpresentation space (or some other space). The decoding system accessesinformation describing the presentation space. The decoding systemdecodes the encoded data using the accessed presentation spaceconfiguration such that the decoded data can be processed by the LFprocessing engine 530 installed in the presentation space. The LFprocessing engine 530 generates and presents content for thepresentation space despite being recorded in a different location.

The network system 556 is any system configured to manage thetransmission of holographic content between systems in a LF sportingevent network 550. For example, the network system 556 may receive arequest for holographic content from a LF display system 500A andfacilitate transmission of the holographic content to the LF displaysystem 500A from the LF generation system 554. The network system 556may also store holographic content, viewer profiles, holographiccontent, etc. for transmission to, and/or storage by, other LF displaysystems 500 in the LF sporting event network 550. The network system 556may also include a LF processing engine 530 that can create holographiccontent as previously described.

The network system 556 may include a digital rights management (DRM)module to manage the digital rights of the holographic content. Forexample, the LF generation system 554 may transmit the holographiccontent to the network system 556 and the DRM module may encrypt theholographic content using a digital encryption format. In otherexamples, the LF generation system 554 encodes recorded light field datainto a holographic content format that can be managed by the DRM module.The network system 556 may provide a key to the digital encryption keyto a LF display system such that each LF display system 500 can decryptand, subsequently, display the holographic content to viewers. Mostgenerally, the network system 556 and/or the LF generation system 554encodes the holographic content and a LF display system may decode theholographic content.

The network system 556 may act as a repository for previously recordedand/or created holographic content. Each piece of holographic contentmay be associated with a transaction fee that, when received, causes thenetwork system 556 to transmit the holographic content to the LF displaysystem 500 that provides the transaction fee. For example, A LF displaysystem 500A may request access to the holographic content via thenetwork 552. The request includes a transaction fee for the holographiccontent. In response, network system 556 transmits the holographiccontent to the LF display system for display to viewers. In otherexamples, the network system 556 can also function as a subscriptionservice for holographic content stored in the network system. In anotherexample, LF generation system 554 is recording light field data of asporting event in real-time and generating holographic contentrepresenting that sporting event. A LF display system 500 transmits arequest for the holographic content to the LF generation system 554. Therequest includes a transaction fee for the holographic content. Inresponse, the LF generation system 554 transmits the holographic contentfor concurrent display on the LF display system 500. The network system556 may act as a mediator in exchanging transaction fees and/or managingholographic content data flow across the network 552. Additionally, insome cases, the network system is capable of modifying holographiccontent such that it is presentable by the LF display system receivingthe holographic content.

In some examples, the network system 556 may be a local distributionhub. That is a network system 556 responsible for distributing aholographic content to LF display system in a local market. For example,the local distribution hub may be a Dallas station responsible fordistributing holographic content to LF display systems 500 in the Dallasarea. In this case, the network system 556 may augment the holographiccontent with additional holographic content, audio content, videocontent, and/or tactile content.

The network 552 represents the communication pathways between systems ina LF sporting event network 550. In one embodiment, the network is theInternet, but can also be any network, including but not limited to alocal area network (LAN), a metropolitan area network (MAN), a wide areanetwork (WAN), a mobile, wired or wireless network, a cloud computingnetwork, a private network, or a virtual private network, and anycombination thereof. In addition, all or some of links can be encryptedusing conventional encryption technologies such as the secure socketslayer (SSL), Secure HTTP and/or virtual private networks (VPNs). Inanother embodiment, the entities can use custom and/or dedicated datacommunications technologies instead of, or in addition to, the onesdescribed above.

III. Example Arenas

FIGS. 6A-9B illustrate several example presentation spaces that maydisplay holographic content using a LF display system (e.g., LF displaysystem 500). The holographic content can be a recorded sporting event, asporting event occurring concurrently in another arena, and/orholographic content that augments a presented sporting event. In thisdisclosure, this content is referred to as holographic sporting eventcontent, or simply holographic sporting content. Within a presentationspace, any number of viewers are located at viewing locations within anynumber of viewing volumes. The LF display system is configured todisplay the holographic sporting content in a holographic object volume(“holographic sporting event volume”) such that viewers in the viewingvolume(s) perceive this holographic sporting content. In this disclosurewe often use the terms “holographic sporting event volume,” and“holographic object volume,” interchangeably, to refer to the region ofspace where the holographic sporting content is projected (e.g. theholographic object volume 255 shown in FIG. 2B). Generally, a LF displaysystem in a venue includes an array of light field modules 210 aroundthe holographic sporting event volume that generate a seamlessmultisided LF display. As such, the holographic sporting event volumemay be the aggregate holographic object volume of the array of LFdisplay modules 210.

FIG. 6 illustrates a side view of a presentation space which is atraditional arena 600 which has been augmented with a LF display system,in accordance with one or more embodiments. In the arena 600, a backwall and the floors are lined with an array 640 of LF display modulessuch that the area above the floor is the holographic sporting eventvolume 610 of the LF display system. In some examples, the arena 600 mayinclude elements in the holographic sporting event volume that allow foreasier presentation of holographic sporting content. For example, aholographic sporting event volume may include a tennis court such thatthe LF display system need not render the court when presenting a tennismatch. The holographic sporting event volume 610 (e.g., holographicobject volume 255) is illustrated as a bounded square for clarity, butthe illustrated holographic sporting event volume 610 is only a portionof the actual holographic sporting event volume. For example, theholographic sporting event volume may extend into the back wall or thefloor 602. In FIG. 6 , the LF display system is an embodiment of the LFdisplay system 500. Further, the LF display modules in the array 640 arean embodiment of the LF display assembly 510.

Here, the arena 600 is three tiered, but could include any number oftiers. Each tier includes a number of viewing locations (e.g., viewinglocation 622A, 622B) for viewers to view holographic sporting eventcontent 630 in the holographic sporting event volume 610. The viewinglocations 622 in each tier are included in a viewing volume (e.g., 620A,620B, and 620C) of the LF display system. The LF display system maydisplay the same, or different, sporting event content 630 to viewers ineach viewing sub-volume 620 (similar to viewing sub-volumes 290 in FIG.2B). For example, as described below, viewers in the viewing locations622A located in the viewing volume 620A of the bottom tier may seedifferent sporting event content 630 than viewers in the viewinglocation 622B in the viewing volume 620B of the middle tier.

In other embodiments, the arena 600 may configured in a differentmanner. For example, the walls and floor may take some other shapeappropriate for the sporting content to be presented. Any of thesurfaces in the arena 600 may be included in the array 640 of LF displaymodules. Additionally, the arena 600 may have additional tiers andviewing volumes and those tiers and viewing volumes may be arranged inany number of configurations.

As a contextual example, the illustrated arena 600 is a presentationspace in San Francisco billed to concurrently show the finals to atennis match “Bumbledon.” Notably, however, the tennis match isoccurring in London, England. The arena in London includes a LFgeneration system (e.g., LF generation system 554) for recording andtransmitting the tennis match as holographic sporting event content 630to other venues via a network (e.g., via network 552). The arena 600 inSan Francisco includes a LF display system configured to display thereceived holographic sporting event content 630 to viewers at viewinglocations 622 in the viewing sub-volumes 620. A transaction fee is paid(e.g., to network system 556 or LF generation system 554) to receive thesporting event content. In various examples, the owner of the arena 600,attendees to the arena 600, a sporting event manager, or any other agentmay pay the fee. The holographic sporting event content 630 allowsviewers in San Francisco to perceive the tennis match as if they wereviewing the match live in the holographic sporting event volume 610 infront of them. This allows viewers in San Francisco to see the livesporting event in a nearly identical way to the viewers in London,without travelling to London.

In some embodiments, the venue 600 charges an entrance fee to see thetennis match live-stream displayed by the LF display system 500. Eachviewing location 622 can have a different entrance fee and, generally,the entrance fee for the bottom viewing volume 620A is more expensivethan the top viewing volume 620C. The LF display system can displaydifferent holographic sporting event content 630 to each viewing volume620 based on the entrance fee for the viewing volumes 620. For example,fully rendering the match in San Francisco may be costlier (e.g.,processing power, energy, etc.) than partially rendering the match. Assuch, a LF processing engine (e.g., LF processing engine 530) may onlydisplay a portion of the sporting content to the viewing volume 620C onthe top tier of the theater while playing all of the sporting content630 to the viewing volume 620A on the bottom tier of the theater. Forexample, when displaying only a portion of the holographic sportingevent content 630, the LF processing engine may only display sportingevent content 630 in a portion of the holographic sporting event volume610 rather than the whole holographic sporting event volume 610, the LFprocessing engine may remove aspects of the holographic sporting eventcontent 630 (e.g., cheerleaders, portions of the fans or spectators,etc.), the LF processing engine 530 may render the holographic sportingevent content 630 at a lower resolution for the top viewing volume 620Cthan the bottom viewing volume 620A, and the like.

Alternatively or additionally, LF processing engine 530 can createholographic content to augment the holographic sporting event content630 based on the entrance fee for the viewing volumes 620. For example,the LF processing engine may create (or access from a network system556) an advertisement to display concurrently with the sporting content630. The advertisement may be based on the information obtained by aviewer profiling system (e.g., viewer profiling system 590) or atracking system (e.g., tracking system 580). For example, the LFprocessing engine may access viewer profiles including viewercharacteristics and responses to holographic content. The LF processingengine accesses an advertisement from the data store (e.g., data store522) that is associated with the characteristics and responses of theviewers and displays that advertisement. The venue may receive paymentfrom the advertiser for displaying the advertisement when displaying theholographic sporting content 630. In another example, rather than anovertly displayed advertisement, LF processing engine can augment theholographic sporting event content 630 with sponsored holographiccontent (e.g., product placements). For example, LF processing enginemay replace courtside advertisements for viewers in London withadvertisements targeting viewers in San Francisco.

FIG. 7 illustrates a cross-section of another arena 700 including a LFdisplay system for displaying sporting content to viewers at viewinglocations in viewing volumes, in accordance with one or moreembodiments. The arena 700 is designed and built to display sportingcontent 730 rather than augmenting an already-existing arena. Asillustrated, FIG. 7 is a cross-section of arena 700. In FIG. 7 , the LFdisplay system 740 is an embodiment of the LF display system 500.

In the illustrated example, the arena 700 resembles a judo presentationspace where the viewing locations 722 circle a fighting ring 702. Thefloor of the fighting ring 702 is covered with an array of LF displaymodules such that the area above the stage forms a holographic sportingevent volume (e.g., sporting event volume 710). The LF display 740presents sporting content 730 in the holographic sporting event volume710 such that viewers in the arena 700 may perceive the holographicsporting event content 730. In the arena 700, the viewing locations 722are positioned with a rake such that the sightline for each viewinglocation allows unobstructed viewing of sporting content 730 from aviewing volume (e.g., viewing volume 720A). Here, the arena includes oneviewing volume which surrounds the fighting ring 702, including viewingvolumes 720A and 720B, such that all the viewers are presented with thesame sporting event. In other configurations there may be more than oneviewing volume.

More generally, the LF display system may have a display surface that issubstantially horizontal, or approximately horizontal. In severalexamples, the LF display system may include a display surface that is(i) at least some part of the floor of the arena, (ii) at least somepart of a fighting ring (e.g., fighting ring 702), gymnastics floor,tennis court, field, or other sporting event surface in an arena and/or(iii) at least some portion of a raised viewing platform in an arena.Other types of horizontal surfaces are also possible. For theseconfigurations, the viewers may be elevated relative to the displaysurface and look downward to view the holographic sporting content thatis projected from the display surface, and the viewers may partially orfully surround the display surface. There are many other configurationsfor a light field display surface, including a vertically-mounteddisplay surface with viewing locations that are arrayed approximately infront of the LF display surface and is described elsewhere in thisdisclosure (450A shown in FIG. 4C, 450B shown in FIG. 4D, and 450C shownin FIG. 4E), as well as a curved LF display surface.

The holographic sporting event volume 710 is illustrated as a boundedsquare for clarity, but the holographic sporting event volume 710 mayonly be a portion of the actual sporting event volume where holographicsporting event objects may be projected. For example, the holographicsporting event volume 710 may extend further towards the top of thearena 700. Additionally, a portion of the sporting event volume 710 andviewing volume 720 (e.g. 720A and 720B) overlap spatially. Whileillustrated as a partial overlap, the holographic sporting event volumeand viewing volume may wholly overlap spatially. The spatial overlap areareas in which viewers may interact with holographic sporting content730 as previously described.

A presentation space can also be a much smaller location. For example,FIG. 8 illustrates a presentation space that also acts as a home theater800 in the living room 802 of a viewer, in accordance with one or moreembodiments. Here, the home theater includes a LF display system 840comprised of an array of LF display modules on one wall. In FIG. 8 , theLF display system 840 is an embodiment of the LF display system 500.

The LF display system 840 may be configured such that the sporting eventvolume (e.g., holographic object volume 255) and the viewing volumewholly overlap within the living room 802. That is, at any viewinglocation (e.g., viewing location 822) within the living room 802 aviewer may view and interact with sporting content 830. While theviewing location 822 is illustrated as a chair, a viewer may movethrough the living room and view the sporting content 830. That is, theviewing location 822 may be the location of the viewer in the livingroom 802.

In some embodiments, the LF display system 840 creates (or modifies)sporting content 830 based on viewer interactions with the holographicsporting content 830. For example, a viewer can fist-bump a performer inthe holographic sporting content 830. In this situation, a viewer mayinteract with a cheerleader and move his hand as if to high-five thecheerleader in the holographic sporting content 830. The tracking system(e.g., tracking system 580) monitors the viewer and identifies (e.g.,via machine hearing, machine vision, neural networks, etc.) that theviewer wishes to fist-bump the performer in the holographic sportingcontent 830. The content generation system (e.g., content generation530) creates (e.g., using machine learning, neural networks, or somecombination thereof) holographic sporting content representing thecheerleader reciprocating the high-five based using the monitored viewerinformation. The tracking system 580 monitors the position of theviewer's hand and when the viewers hand and cheerleaders hand spatiallyoverlap, the sensory feedback assembly (e.g., sensor feedback assembly570) creates a sensation for the viewer that he has high-fived theperformer in the holographic sporting content 830. This may beaccomplished with a focused ultrasonic projection system which generatesa volumetric tactile surface which generates the sensation of ahigh-five being returned to the viewer. In an embodiment, the ultrasonicprojection system is integrated into the LF display system as a dualenergy display surface, as previously described.

In some cases, the viewer may interact with network system (e.g.,network system 556) to procure sporting content 830 for their livingroom 802. Continuing the example described in regard to FIG. 6 , andreferring to the LF sporting event distribution network shown in FIG.5B, a viewer may send a transaction fee to network system 556 and the LFgeneration system 554 sends the presentation space 800 the holographicsporting content 830 of the tennis match being played in London.

FIG. 9 illustrates a home theater 900 in the living room 902 of aviewer, in accordance with one or more embodiments. Here, the hometheater includes a LF display system 940 built into a table 904, with anarray of LF display modules (“LF array”) on the top surface of the table904. While the illustrated home theater is built into a table, it couldbe built into any other element of a living room having a surface. TheLF array 920 is configured to present holographic sporting content 930to viewers in viewing locations that can view the top surface of thetable 904. In the illustrated example, an example viewing location 922is a chair positioned such that a viewer sitting in the chair canperceive holographic content presented by the LF array 920. Aspreviously described, viewers in other viewing locations may alsoperceive the holographic sporting content 930 presented by the LF array920. In FIG. 10 , the LF array 920 is an embodiment of the LF displayassembly 510. Further, the LF display system 940 is an embodiment of theLF display system 500.

The LF array 920 presents holographic sporting content 930 of a sportingevent such that it appears on the top surface of the table 904. A viewerin the viewing location 922 may be able to perceive and interact withthe presented holographic content 930. In this example, the LF displaysystem inputs holographic content from a network system 556, convertsthe holographic content for display on the top of the table 904, andpresents the holographic content 930 to viewers in one or more viewingvolumes of the LF display system 940.

In some embodiments, a viewer can interact with the LF display system tochange the presented holographic sporting content 930. For example, theLF display system can be configured to receive auditory cues, visualcues, etc. and change the holographic content 930 in response. As anillustration, a viewer in the living room 902 can state “Pause Match,”the LF display system records the audio, recognizes the audio, andpauses playback of the holographic sporting content 930 in response.Similarly, a viewer in the living room 902 can interact with theholographic content 930 to rotate the view displayed by the table. As anillustration, a viewer can touch a player in the presented holographiccontent 930 with one hand and make a rotation gesture with the otherhand. In this case, the LF display system captures an image of theviewer making the gesture, recognizes the gesture, and rotates theholographic content in response.

IV. Displaying Sporting Event Content to Viewers in an Arena

FIG. 10 is a flow diagram of a method 1000 for displaying holographiccontent to viewers in a sports event presentation space (e.g., arena600) in the context of a LF sports network (e.g., LF sports network550). The method 1000 may include additional or fewer steps and thesteps may occur in a different order. Further, various steps, orcombinations of steps, can be repeated any number of times duringexecution of the method.

To begin, an arena including a LF display system (e.g., LF displaysystem 500) transmits 1010 a request for a live-stream (or previouslyrecorded) of holographic sporting event content (e.g., holographicsporting event content 630) to a network system (e.g., network system556) system via a network (e.g., network 552). The request may include atransaction fee sufficient for payment to display the holographicsporting event content.

A LF generation system (e.g., LF generation system 554) records the LFdata of a live sporting event and transmits the correspondingholographic sporting event content to the network system. The networksystem transmits the holographic sporting event content to the LFdisplay system such that it can be displayed at approximately the sametime as it is being recorded by the LF generation system. In otherembodiments, the network system (e.g. network system 556) may transmitpre-recorded holographic sporting event content.

The LF display system receives 1020 the holographic sporting eventcontent from the network system via the network.

The LF display system determines 1030 a configuration of the LF displaysystem and/or the presentation space in the arena. For example, the LFdisplay system may access a configuration file including a number ofparameters describing the HW configuration of the LF display, includingthe resolution, projected rays per degree, fields-of-view, deflectionangles on the display surface, or a dimensionality of the LF displaysurface. The configuration file may also contain information about thegeometrical orientation of the LF display assembly, including the numberof LF display panels, relative orientation, width, height, and thelayout of the LF display panels. Further, the configuration file maycontain configuration parameters of the performance venue, includingholographic object volumes, viewing volumes, and a location of theaudience relative to the display panels.

To illustrate through an example, the LF display system determines 1030viewing volumes (e.g., viewing volume 620A, 620B, and 620C) fordisplaying the holographic sporting event content. For example, the LFdisplay system 500 may access information in the LF display systemdescribing the layout, geometric configuration, and/or hardwareconfiguration of the presentation space (e.g., arena 600). Toillustrate, the layout may include the locations, separations, and sizesof viewing locations (e.g., viewing locations 622) in the presentationspace. As such, LF display system may determine that viewing locationsin the first tier of the presentation space are in a first viewingvolume, viewing locations in the second tier of the venue are in asecond viewing volume, and viewing locations in the third tier of thevenue are in a third viewing volume. In various other embodiments, a LFdisplay system may determine any number and configuration of viewingvolumes at any location within a venue.

The LF display system generates 1040 the holographic content (and othersensory content) for presenting on the LF display system, based on thehardware configuration of the LF display system within the performancevenue, and the particular layout and configuration of the presentationspace. Determining the holographic sporting event content for displaycan include appropriately rendering the holographic sporting eventcontent for the presentation space or viewing volumes. For example, theLF display system may: (i) augment the holographic sporting eventcontent for display to the viewing volume in the third tier with anadvertisement and remove aspects of the live performance from theholographic sporting event content, (ii) decrease the fidelity of theholographic sporting event content for display to the viewing volume inthe second tier of the presentation space, and (iii) fully renderholographic sporting event content for display to the viewing volume inthe third tier of the presentation space.

The LF display system presents 1050 the holographic sporting eventcontent in the holographic sporting event volume of the presentationspace such that viewers at viewing locations in each viewing volumeperceive the appropriate holographic sporting event content. That is,viewers in the top viewing volume perceive the holographic sportingevent content with an advertisement, viewers in the middle viewingvolume perceive the holographic sporting event content at lowerresolution, and viewers in the bottom viewing volume perceive the fullyrendered holographic sporting event content.

The LF display system may determine information about viewers in theviewing volumes while the viewers view the holographic sporting eventcontent at any time. For example, the tracking system may monitor thefacial responses of viewers in the viewing volumes and the viewerprofiling system may access information regarding characteristics of theviewers in the viewing volumes.

The LF display system may create (or modify) holographic sporting eventcontent for concurrent display based on the determined information. Forexample, the LF processing engine may create a light show for concurrentdisplay by the LF display system based on determined information thatthe viewers enjoy fireworks or electronic music festivals.

V. Additional Configuration Information

The foregoing description of the embodiments of the disclosure has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the disclosure be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

1. A light field (LF) sports content network system comprising: a lightfield recording assembly configured to record one or more types ofenergy representing a sporting event at a sporting arena; a processingengine configured to convert the recorded energy into a holographiccontent livestream representing the sporting event; and a networkinterface configured to transmit the holographic content livestream toone or more LF display systems via a network.
 2. The LF sports contentnetwork system of claim 1, wherein the network is any of: a cable systemnetwork; an IP-based network; and a satellite network.
 3. The LF sportscontent network system of claim 1, wherein the processing engine isconfigured to add additional content to the holographic contentlivestream.
 4. The LF sports content network system of claim 3, whereinthe additional content is audio content, the audio content presentedsimultaneously to the holographic content livestream by a LF displaysystem.
 5. The LF sports content network system of claim 4, wherein theaudio content is any of: an announcer; a music track; a jingle; a soundeffect; or a translation.
 6. The LF sports content network system ofclaim 3, wherein the additional content is presented simultaneously tothe holographic content livestream by a LF display system.
 7. The LFsports content network system of claim 6, wherein the additionalholographic content is any of: an overlay for a sporting area in thesporting area; an informational overlay; and an augmentation ofholographic content included in the holographic content livestream. 8.The LF sports content network system of claim 3, wherein the additionalcontent is an advertisement, the advertisement presented simultaneouslyto the holographic content livestream.
 9. The LF sports content networksystem of claim 8, wherein the advertisement includes any of:holographic content; video content; tactile content.
 10. The LF sportscontent network system of claim 8, wherein the advertisement presentedby the LF display system is based on any of: a location of the LFdisplay system; the sporting arena; a sponsor of the sporting event; aconfiguration of the LF display system; a transaction fee provided bythe LF display system to the LF sports content network system for theholographic content live stream; a type of the sporting eventrepresented the holographic content livestream; a time of day that theholographic content livestream is presented; and a player, a team, aparticipant, a contestant, a contender, or a coach participating in thesporting event.
 11. The LF sports content network system of claim 6,wherein the additional content is tactile content.
 12. The LF sportscontent network system of claim 1, wherein the processing enginecomprises an encoder to encode the holographic content livestream to anencoded holographic content livestream, and the encoded holographiccontent livestream is transmitted via the network interface.
 13. The LFsports content network system of claim 12, wherein the encodedholographic content livestream is configured to be decoded by a decoderof a LF display system.
 14. The LF sports content network system ofclaim 12, wherein the encoder encodes the encoded holographic livestreamin approximately real time.
 15. The LF sports content network system ofclaim 12, wherein: the encoder employs a proprietary codec to encode theholographic content livestream, and a decoder employing the proprietarycodec decodes the encoded holographic content livestream.
 16. The LFsports content network system of claim 12, wherein the holographiccontent livestream is a first data format, and the encoded holographiccontent livestream is a second data format.
 17. The LF sports contentnetwork system of claim 16, wherein: the first data format is arasterized data format, and the second data format is a vectorized dataformat.
 18. The LF sports content network system of claim 1, wherein:the network interface transmits the holographic content livestream to alocal holographic content distribution hub, and the local holographiccontent distribution hub transmits the holographic content livestream toat least one LF display system of the one or more LF display systems.19. The LF sports content network system of claim 18, wherein the localholographic content distribution hub adds additional content toholographic content livestream.
 20. The LF sports content network systemof claim 19, wherein the additional content includes any of: localcontent; local advertisements; holographic content; audio content; videocontent; and tactile content. 21.-97. (canceled)